2024-04-18T21:38:13.116 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-18T21:38:13.117 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549 --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} -- /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/orig.config.yaml 2024-04-18T21:38:13.161 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T21:38:13.263 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549 branch: squid description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} email: ceph-qa@ceph.com first_in_suite: false job_id: '7662549' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: 1a6a3e742fd81c6d70be70552499bff767652c32 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1a6a3e742fd81c6d70be70552499bff767652c32 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d0bf63dd179c05751e4be49477389aa290c73089 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - mon.a - - osd.4 - osd.5 - osd.6 - osd.7 - mon.b - mgr.x - - osd.8 - osd.9 - osd.10 - osd.11 - mon.c seed: 6512 sha1: 1a6a3e742fd81c6d70be70552499bff767652c32 sleep_before_teardown: 0 subset: 28391/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d0bf63dd179c05751e4be49477389aa290c73089/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d0bf63dd179c05751e4be49477389aa290c73089 targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8BTLlHsQIqVUe8iE//YUlOuJdavNucK3J/5KjI/Uys9To7LIC1sYGPUZgtrccolcjJ/hhbDNlQ+CdWV45drZE= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMvqbcsyJ/fgp+yQJNDvEpdvtoJWniApzcFsC+8Ew8QoTjwm+ZtaV4Mgm9122E3dwtsqQeP68oLn/q2qpYAlPNA= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNt/faOsP8WklJZIcxe6MukZBvmG9a554VAh3KDl1EZAoFU77pNllWrJYYQEqlebE6ssaNI90pAxMYpoX94csXw= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: crush-failure-domain: osd k: 4 l: 3 m: 2 name: lrcprofile plugin: lrc objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-18_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-18T21:38:13.263 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d0bf63dd179c05751e4be49477389aa290c73089/qa; will attempt to use it 2024-04-18T21:38:13.264 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d0bf63dd179c05751e4be49477389aa290c73089/qa/tasks 2024-04-18T21:38:13.264 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-18T21:38:13.265 INFO:teuthology.task.internal:Checking packages... 2024-04-18T21:38:13.289 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '1a6a3e742fd81c6d70be70552499bff767652c32' 2024-04-18T21:38:13.289 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-18T21:38:13.289 INFO:teuthology.packaging:ref: None 2024-04-18T21:38:13.289 INFO:teuthology.packaging:tag: None 2024-04-18T21:38:13.289 INFO:teuthology.packaging:branch: squid 2024-04-18T21:38:13.289 INFO:teuthology.packaging:sha1: 1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-18T21:38:13.290 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-18T21:38:13.502 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:38:13.504 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-18T21:38:13.513 INFO:teuthology.task.internal:no buildpackages task found 2024-04-18T21:38:13.513 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-18T21:38:13.555 INFO:teuthology.task.internal:Saving configuration 2024-04-18T21:38:13.571 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-18T21:38:13.588 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-18T21:38:13.614 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 21:30:37.754884', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAFtIBvCvO9XmhcPW4LdmXVwYlpVt+wm9yEU8M8fs75p'} 2024-04-18T21:38:13.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 21:30:37.752048', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC2r1iHwBziQuUxLrTFaU2zsUH3voolhBh8PdX5nO98j'} 2024-04-18T21:38:13.660 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 21:30:37.753508', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILt/kFDyjKo0nnz3bq/nDc8p4owtbSLjUECwosNW99d4'} 2024-04-18T21:38:13.660 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-18T21:38:13.666 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2024-04-18T21:38:13.667 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2024-04-18T21:38:13.667 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2024-04-18T21:38:13.667 INFO:teuthology.run_tasks:Running task console_log... 2024-04-18T21:38:13.781 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7dc53da280>, signals=[15]) 2024-04-18T21:38:13.781 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-18T21:38:13.788 INFO:teuthology.task.internal:Opening connections... 2024-04-18T21:38:13.788 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:38:13.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:38:13.866 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:38:13.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:38:13.946 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:38:13.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:38:14.026 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-18T21:38:14.034 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-04-18T21:38:14.040 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-04-18T21:38:14.040 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-04-18T21:38:14.087 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-18T21:38:14.087 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="Ubuntu" 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="22.04" 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_CODENAME=jammy 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:ID=ubuntu 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE=debian 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-18T21:38:14.088 INFO:teuthology.orchestra.run.smithi097.stdout:UBUNTU_CODENAME=jammy 2024-04-18T21:38:14.089 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-04-18T21:38:14.118 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-04-18T21:38:14.124 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-04-18T21:38:14.124 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-04-18T21:38:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-18T21:38:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Ubuntu" 2024-04-18T21:38:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="22.04" 2024-04-18T21:38:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-18T21:38:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_CODENAME=jammy 2024-04-18T21:38:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:ID=ubuntu 2024-04-18T21:38:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE=debian 2024-04-18T21:38:14.171 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-18T21:38:14.171 INFO:teuthology.orchestra.run.smithi104.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-18T21:38:14.171 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-18T21:38:14.171 INFO:teuthology.orchestra.run.smithi104.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-18T21:38:14.171 INFO:teuthology.orchestra.run.smithi104.stdout:UBUNTU_CODENAME=jammy 2024-04-18T21:38:14.171 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-04-18T21:38:14.209 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-04-18T21:38:14.215 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-04-18T21:38:14.216 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-04-18T21:38:14.267 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-18T21:38:14.267 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Ubuntu" 2024-04-18T21:38:14.267 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="22.04" 2024-04-18T21:38:14.267 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-18T21:38:14.267 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_CODENAME=jammy 2024-04-18T21:38:14.267 INFO:teuthology.orchestra.run.smithi183.stdout:ID=ubuntu 2024-04-18T21:38:14.268 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE=debian 2024-04-18T21:38:14.268 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-18T21:38:14.268 INFO:teuthology.orchestra.run.smithi183.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-18T21:38:14.268 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-18T21:38:14.268 INFO:teuthology.orchestra.run.smithi183.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-18T21:38:14.268 INFO:teuthology.orchestra.run.smithi183.stdout:UBUNTU_CODENAME=jammy 2024-04-18T21:38:14.268 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-04-18T21:38:14.293 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-18T21:38:14.302 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-18T21:38:14.363 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-18T21:38:14.363 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-04-18T21:38:14.366 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-04-18T21:38:14.368 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-04-18T21:38:14.374 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-18T21:38:14.387 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-18T21:38:14.388 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-04-18T21:38:14.413 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-04-18T21:38:14.415 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-04-18T21:38:14.428 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-18T21:38:14.491 INFO:teuthology.run_tasks:Running task kernel... 2024-04-18T21:38:14.510 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-18T21:38:14.510 INFO:teuthology.task.kernel:config {'osd.0': {'kdb': 1, 'sha1': 'distro'}, 'osd.4': {'kdb': 1, 'sha1': 'distro'}, 'osd.8': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-18T21:38:14.510 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-18T21:38:14.511 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-18T21:38:14.511 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-18T21:38:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:38:14.516 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-04-18T21:38:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:38:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:38:14.519 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-04-18T21:38:14.519 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-04-18T21:38:14.562 INFO:teuthology.orchestra.run.smithi104.stdout:5.15.0-56-generic 2024-04-18T21:38:14.563 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.15.0-56-generic 2024-04-18T21:38:14.563 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2024-04-18T21:38:14.568 INFO:teuthology.orchestra.run.smithi097.stdout:5.15.0-56-generic 2024-04-18T21:38:14.568 INFO:teuthology.task.kernel:Running kernel on smithi097: 5.15.0-56-generic 2024-04-18T21:38:14.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get clean 2024-04-18T21:38:14.570 INFO:teuthology.orchestra.run.smithi183.stdout:5.15.0-56-generic 2024-04-18T21:38:14.570 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.15.0-56-generic 2024-04-18T21:38:14.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo apt-get clean 2024-04-18T21:38:14.694 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-04-18T21:38:14.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get update 2024-04-18T21:38:14.706 DEBUG:teuthology.orchestra.run.smithi183:> sudo apt-get update 2024-04-18T21:38:14.827 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:38:14.844 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-18T21:38:14.849 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:38:14.857 INFO:teuthology.orchestra.run.smithi097.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-18T21:38:14.860 INFO:teuthology.orchestra.run.smithi183.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-18T21:38:14.922 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-18T21:38:14.937 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-18T21:38:14.942 INFO:teuthology.orchestra.run.smithi183.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-18T21:38:14.949 INFO:teuthology.orchestra.run.smithi183.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-18T21:38:14.965 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:38:15.047 INFO:teuthology.orchestra.run.smithi097.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-18T21:38:15.061 INFO:teuthology.orchestra.run.smithi183.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [611 kB] 2024-04-18T21:38:15.065 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [611 kB] 2024-04-18T21:38:15.079 INFO:teuthology.orchestra.run.smithi097.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-18T21:38:15.149 INFO:teuthology.orchestra.run.smithi183.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,562 kB] 2024-04-18T21:38:15.150 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,562 kB] 2024-04-18T21:38:15.180 INFO:teuthology.orchestra.run.smithi097.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,346 kB] 2024-04-18T21:38:15.199 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [298 kB] 2024-04-18T21:38:15.205 INFO:teuthology.orchestra.run.smithi183.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [298 kB] 2024-04-18T21:38:15.207 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-18T21:38:15.209 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,754 kB] 2024-04-18T21:38:15.216 INFO:teuthology.orchestra.run.smithi183.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-18T21:38:15.216 INFO:teuthology.orchestra.run.smithi183.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,754 kB] 2024-04-18T21:38:15.251 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-18T21:38:15.251 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [295 kB] 2024-04-18T21:38:15.257 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T21:38:15.260 INFO:teuthology.orchestra.run.smithi183.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-18T21:38:15.261 INFO:teuthology.orchestra.run.smithi183.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [295 kB] 2024-04-18T21:38:15.265 INFO:teuthology.orchestra.run.smithi183.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T21:38:15.265 INFO:teuthology.orchestra.run.smithi183.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-18T21:38:15.271 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-18T21:38:15.275 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [237 kB] 2024-04-18T21:38:15.275 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,074 kB] 2024-04-18T21:38:15.284 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-18T21:38:15.285 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,670 kB] 2024-04-18T21:38:15.292 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-04-18T21:38:15.294 INFO:teuthology.orchestra.run.smithi183.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,074 kB] 2024-04-18T21:38:15.297 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-18T21:38:15.297 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-18T21:38:15.298 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-18T21:38:15.307 INFO:teuthology.orchestra.run.smithi183.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-04-18T21:38:15.310 INFO:teuthology.orchestra.run.smithi183.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-18T21:38:15.311 INFO:teuthology.orchestra.run.smithi183.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-18T21:38:15.311 INFO:teuthology.orchestra.run.smithi183.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-18T21:38:15.314 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-18T21:38:15.314 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-18T21:38:15.314 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-18T21:38:15.315 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-18T21:38:15.315 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-18T21:38:15.316 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-18T21:38:15.325 INFO:teuthology.orchestra.run.smithi183.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-18T21:38:15.325 INFO:teuthology.orchestra.run.smithi183.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-18T21:38:15.325 INFO:teuthology.orchestra.run.smithi183.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-18T21:38:15.325 INFO:teuthology.orchestra.run.smithi183.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-18T21:38:15.326 INFO:teuthology.orchestra.run.smithi183.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-18T21:38:15.328 INFO:teuthology.orchestra.run.smithi097.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [34.1 kB] 2024-04-18T21:38:15.329 INFO:teuthology.orchestra.run.smithi097.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [280 kB] 2024-04-18T21:38:15.334 INFO:teuthology.orchestra.run.smithi097.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T21:38:15.334 INFO:teuthology.orchestra.run.smithi097.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-18T21:38:15.337 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-18T21:38:15.338 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-18T21:38:15.338 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-18T21:38:15.338 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-18T21:38:15.345 INFO:teuthology.orchestra.run.smithi097.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-18T21:38:15.345 INFO:teuthology.orchestra.run.smithi183.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-18T21:38:15.346 INFO:teuthology.orchestra.run.smithi183.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-18T21:38:15.346 INFO:teuthology.orchestra.run.smithi183.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-18T21:38:15.347 INFO:teuthology.orchestra.run.smithi183.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-18T21:38:15.347 INFO:teuthology.orchestra.run.smithi183.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-18T21:38:15.357 INFO:teuthology.orchestra.run.smithi097.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-18T21:38:15.359 INFO:teuthology.orchestra.run.smithi097.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-18T21:38:15.360 INFO:teuthology.orchestra.run.smithi097.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-18T21:38:15.367 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-18T21:38:15.377 INFO:teuthology.orchestra.run.smithi097.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-18T21:38:15.377 INFO:teuthology.orchestra.run.smithi097.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-18T21:38:15.377 INFO:teuthology.orchestra.run.smithi097.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-18T21:38:15.400 INFO:teuthology.orchestra.run.smithi183.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,346 kB] 2024-04-18T21:38:15.411 INFO:teuthology.orchestra.run.smithi097.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-18T21:38:15.549 INFO:teuthology.orchestra.run.smithi097.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,562 kB] 2024-04-18T21:38:15.649 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,346 kB] 2024-04-18T21:38:15.807 INFO:teuthology.orchestra.run.smithi183.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-18T21:38:15.815 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [237 kB] 2024-04-18T21:38:15.825 INFO:teuthology.orchestra.run.smithi183.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [237 kB] 2024-04-18T21:38:15.827 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-18T21:38:15.828 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,670 kB] 2024-04-18T21:38:15.834 INFO:teuthology.orchestra.run.smithi183.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-18T21:38:15.834 INFO:teuthology.orchestra.run.smithi183.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,670 kB] 2024-04-18T21:38:15.856 INFO:teuthology.orchestra.run.smithi097.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [611 kB] 2024-04-18T21:38:15.895 INFO:teuthology.orchestra.run.smithi097.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [298 kB] 2024-04-18T21:38:15.909 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [34.1 kB] 2024-04-18T21:38:15.911 INFO:teuthology.orchestra.run.smithi097.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-18T21:38:15.911 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [280 kB] 2024-04-18T21:38:15.911 INFO:teuthology.orchestra.run.smithi097.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,754 kB] 2024-04-18T21:38:15.918 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T21:38:15.919 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-18T21:38:15.926 INFO:teuthology.orchestra.run.smithi183.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [34.1 kB] 2024-04-18T21:38:15.927 INFO:teuthology.orchestra.run.smithi183.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [280 kB] 2024-04-18T21:38:15.934 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-18T21:38:15.938 INFO:teuthology.orchestra.run.smithi183.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T21:38:15.938 INFO:teuthology.orchestra.run.smithi183.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-18T21:38:15.970 INFO:teuthology.orchestra.run.smithi183.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-18T21:38:15.973 INFO:teuthology.orchestra.run.smithi097.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-18T21:38:15.974 INFO:teuthology.orchestra.run.smithi097.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [295 kB] 2024-04-18T21:38:15.981 INFO:teuthology.orchestra.run.smithi097.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T21:38:16.118 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-18T21:38:16.145 INFO:teuthology.orchestra.run.smithi097.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,074 kB] 2024-04-18T21:38:16.146 INFO:teuthology.orchestra.run.smithi183.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-18T21:38:16.524 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-18T21:38:16.530 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-18T21:38:16.531 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-18T21:38:16.544 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-18T21:38:16.547 INFO:teuthology.orchestra.run.smithi104.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-18T21:38:16.554 INFO:teuthology.orchestra.run.smithi183.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-18T21:38:16.560 INFO:teuthology.orchestra.run.smithi183.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-18T21:38:16.561 INFO:teuthology.orchestra.run.smithi183.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-18T21:38:16.575 INFO:teuthology.orchestra.run.smithi183.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-18T21:38:16.578 INFO:teuthology.orchestra.run.smithi183.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-18T21:38:16.782 INFO:teuthology.orchestra.run.smithi097.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-18T21:38:16.819 INFO:teuthology.orchestra.run.smithi097.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-04-18T21:38:16.832 INFO:teuthology.orchestra.run.smithi097.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-18T21:38:16.833 INFO:teuthology.orchestra.run.smithi097.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-18T21:38:16.836 INFO:teuthology.orchestra.run.smithi097.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-18T21:38:16.836 INFO:teuthology.orchestra.run.smithi097.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-18T21:38:16.837 INFO:teuthology.orchestra.run.smithi097.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-18T21:38:16.837 INFO:teuthology.orchestra.run.smithi097.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-18T21:38:16.864 INFO:teuthology.orchestra.run.smithi097.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-18T21:38:16.867 INFO:teuthology.orchestra.run.smithi097.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-18T21:38:17.032 INFO:teuthology.orchestra.run.smithi097.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-18T21:38:17.114 INFO:teuthology.orchestra.run.smithi097.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-18T21:38:17.196 INFO:teuthology.orchestra.run.smithi097.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-18T21:38:17.277 INFO:teuthology.orchestra.run.smithi097.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-18T21:38:17.307 INFO:teuthology.orchestra.run.smithi097.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-18T21:38:20.827 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 12.9 MB in 2s (7,210 kB/s) 2024-04-18T21:38:20.896 INFO:teuthology.orchestra.run.smithi183.stdout:Fetched 12.9 MB in 2s (7,138 kB/s) 2024-04-18T21:38:21.576 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 12.9 MB in 3s (5,106 kB/s) 2024-04-18T21:38:22.077 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-04-18T21:38:22.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T21:38:22.156 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-04-18T21:38:22.245 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2024-04-18T21:38:22.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T21:38:22.322 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2024-04-18T21:38:22.355 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-04-18T21:38:22.355 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-04-18T21:38:22.521 INFO:teuthology.orchestra.run.smithi183.stdout:Building dependency tree... 2024-04-18T21:38:22.521 INFO:teuthology.orchestra.run.smithi183.stdout:Reading state information... 2024-04-18T21:38:22.530 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:38:22.530 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:38:22.530 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:38:22.530 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:38:22.530 INFO:teuthology.orchestra.run.smithi104.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-18T21:38:22.531 INFO:teuthology.orchestra.run.smithi104.stdout: python2.7-minimal 2024-04-18T21:38:22.531 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:38:22.532 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-04-18T21:38:22.532 INFO:teuthology.orchestra.run.smithi104.stdout: linux-generic linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T21:38:22.532 INFO:teuthology.orchestra.run.smithi104.stdout: linux-headers-generic linux-image-5.15.0-102-generic 2024-04-18T21:38:22.532 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-5.15.0-102-generic linux-modules-extra-5.15.0-102-generic 2024-04-18T21:38:22.533 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-04-18T21:38:22.533 INFO:teuthology.orchestra.run.smithi104.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-18T21:38:22.533 INFO:teuthology.orchestra.run.smithi104.stdout:Recommended packages: 2024-04-18T21:38:22.534 INFO:teuthology.orchestra.run.smithi104.stdout: thermald 2024-04-18T21:38:22.581 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-04-18T21:38:22.581 INFO:teuthology.orchestra.run.smithi104.stdout: linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T21:38:22.581 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.15.0-102-generic linux-modules-5.15.0-102-generic 2024-04-18T21:38:22.581 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-extra-5.15.0-102-generic 2024-04-18T21:38:22.583 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages will be upgraded: 2024-04-18T21:38:22.583 INFO:teuthology.orchestra.run.smithi104.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-18T21:38:22.680 INFO:teuthology.orchestra.run.smithi104.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T21:38:22.680 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 113 MB of archives. 2024-04-18T21:38:22.680 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-18T21:38:22.680 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-102-generic amd64 5.15.0-102.112 [22.7 MB] 2024-04-18T21:38:22.696 INFO:teuthology.orchestra.run.smithi183.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:38:22.696 INFO:teuthology.orchestra.run.smithi183.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:38:22.696 INFO:teuthology.orchestra.run.smithi183.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:38:22.696 INFO:teuthology.orchestra.run.smithi183.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:38:22.697 INFO:teuthology.orchestra.run.smithi183.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-18T21:38:22.697 INFO:teuthology.orchestra.run.smithi183.stdout: python2.7-minimal 2024-04-18T21:38:22.697 INFO:teuthology.orchestra.run.smithi183.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:38:22.697 INFO:teuthology.orchestra.run.smithi183.stdout:The following additional packages will be installed: 2024-04-18T21:38:22.698 INFO:teuthology.orchestra.run.smithi183.stdout: linux-generic linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T21:38:22.698 INFO:teuthology.orchestra.run.smithi183.stdout: linux-headers-generic linux-image-5.15.0-102-generic 2024-04-18T21:38:22.698 INFO:teuthology.orchestra.run.smithi183.stdout: linux-modules-5.15.0-102-generic linux-modules-extra-5.15.0-102-generic 2024-04-18T21:38:22.699 INFO:teuthology.orchestra.run.smithi183.stdout:Suggested packages: 2024-04-18T21:38:22.699 INFO:teuthology.orchestra.run.smithi183.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-18T21:38:22.699 INFO:teuthology.orchestra.run.smithi183.stdout:Recommended packages: 2024-04-18T21:38:22.699 INFO:teuthology.orchestra.run.smithi183.stdout: thermald 2024-04-18T21:38:22.755 INFO:teuthology.orchestra.run.smithi183.stdout:The following NEW packages will be installed: 2024-04-18T21:38:22.755 INFO:teuthology.orchestra.run.smithi183.stdout: linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T21:38:22.755 INFO:teuthology.orchestra.run.smithi183.stdout: linux-image-5.15.0-102-generic linux-modules-5.15.0-102-generic 2024-04-18T21:38:22.755 INFO:teuthology.orchestra.run.smithi183.stdout: linux-modules-extra-5.15.0-102-generic 2024-04-18T21:38:22.756 INFO:teuthology.orchestra.run.smithi183.stdout:The following packages will be upgraded: 2024-04-18T21:38:22.756 INFO:teuthology.orchestra.run.smithi183.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-18T21:38:22.864 INFO:teuthology.orchestra.run.smithi183.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T21:38:22.864 INFO:teuthology.orchestra.run.smithi183.stdout:Need to get 113 MB of archives. 2024-04-18T21:38:22.864 INFO:teuthology.orchestra.run.smithi183.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-18T21:38:22.864 INFO:teuthology.orchestra.run.smithi183.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-102-generic amd64 5.15.0-102.112 [22.7 MB] 2024-04-18T21:38:23.060 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2024-04-18T21:38:23.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T21:38:23.132 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2024-04-18T21:38:23.321 INFO:teuthology.orchestra.run.smithi097.stdout:Building dependency tree... 2024-04-18T21:38:23.322 INFO:teuthology.orchestra.run.smithi097.stdout:Reading state information... 2024-04-18T21:38:23.475 INFO:teuthology.orchestra.run.smithi097.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:38:23.475 INFO:teuthology.orchestra.run.smithi097.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:38:23.475 INFO:teuthology.orchestra.run.smithi097.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:38:23.476 INFO:teuthology.orchestra.run.smithi097.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:38:23.476 INFO:teuthology.orchestra.run.smithi097.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-18T21:38:23.476 INFO:teuthology.orchestra.run.smithi097.stdout: python2.7-minimal 2024-04-18T21:38:23.476 INFO:teuthology.orchestra.run.smithi097.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:38:23.477 INFO:teuthology.orchestra.run.smithi097.stdout:The following additional packages will be installed: 2024-04-18T21:38:23.477 INFO:teuthology.orchestra.run.smithi097.stdout: linux-generic linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T21:38:23.477 INFO:teuthology.orchestra.run.smithi097.stdout: linux-headers-generic linux-image-5.15.0-102-generic 2024-04-18T21:38:23.478 INFO:teuthology.orchestra.run.smithi097.stdout: linux-modules-5.15.0-102-generic linux-modules-extra-5.15.0-102-generic 2024-04-18T21:38:23.479 INFO:teuthology.orchestra.run.smithi097.stdout:Suggested packages: 2024-04-18T21:38:23.479 INFO:teuthology.orchestra.run.smithi097.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-18T21:38:23.479 INFO:teuthology.orchestra.run.smithi097.stdout:Recommended packages: 2024-04-18T21:38:23.479 INFO:teuthology.orchestra.run.smithi097.stdout: thermald 2024-04-18T21:38:23.529 INFO:teuthology.orchestra.run.smithi097.stdout:The following NEW packages will be installed: 2024-04-18T21:38:23.529 INFO:teuthology.orchestra.run.smithi097.stdout: linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T21:38:23.529 INFO:teuthology.orchestra.run.smithi097.stdout: linux-image-5.15.0-102-generic linux-modules-5.15.0-102-generic 2024-04-18T21:38:23.530 INFO:teuthology.orchestra.run.smithi097.stdout: linux-modules-extra-5.15.0-102-generic 2024-04-18T21:38:23.531 INFO:teuthology.orchestra.run.smithi097.stdout:The following packages will be upgraded: 2024-04-18T21:38:23.531 INFO:teuthology.orchestra.run.smithi097.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-18T21:38:23.668 INFO:teuthology.orchestra.run.smithi097.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T21:38:23.668 INFO:teuthology.orchestra.run.smithi097.stdout:Need to get 113 MB of archives. 2024-04-18T21:38:23.668 INFO:teuthology.orchestra.run.smithi097.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-18T21:38:23.668 INFO:teuthology.orchestra.run.smithi097.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-102-generic amd64 5.15.0-102.112 [22.7 MB] 2024-04-18T21:38:25.343 INFO:teuthology.orchestra.run.smithi183.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-102-generic amd64 5.15.0-102.112 [11.5 MB] 2024-04-18T21:38:25.360 INFO:teuthology.orchestra.run.smithi097.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-102-generic amd64 5.15.0-102.112 [11.5 MB] 2024-04-18T21:38:25.552 INFO:teuthology.orchestra.run.smithi097.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-102-generic amd64 5.15.0-102.112 [63.9 MB] 2024-04-18T21:38:26.353 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-102-generic amd64 5.15.0-102.112 [11.5 MB] 2024-04-18T21:38:26.504 INFO:teuthology.orchestra.run.smithi183.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-102-generic amd64 5.15.0-102.112 [63.9 MB] 2024-04-18T21:38:26.892 INFO:teuthology.orchestra.run.smithi097.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.102.99 [1,700 B] 2024-04-18T21:38:26.893 INFO:teuthology.orchestra.run.smithi097.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.102.99 [2,512 B] 2024-04-18T21:38:26.893 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102 all 5.15.0-102.112 [12.4 MB] 2024-04-18T21:38:27.152 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102-generic amd64 5.15.0-102.112 [2,877 kB] 2024-04-18T21:38:27.191 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.102.99 [2,360 B] 2024-04-18T21:38:27.350 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-102-generic amd64 5.15.0-102.112 [63.9 MB] 2024-04-18T21:38:27.588 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 113 MB in 4s (31.0 MB/s) 2024-04-18T21:38:27.779 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-modules-5.15.0-102-generic. 2024-04-18T21:38:28.936 INFO:teuthology.orchestra.run.smithi183.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.102.99 [1,700 B] 2024-04-18T21:38:28.936 INFO:teuthology.orchestra.run.smithi183.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.102.99 [2,512 B] 2024-04-18T21:38:28.936 INFO:teuthology.orchestra.run.smithi183.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102 all 5.15.0-102.112 [12.4 MB] 2024-04-18T21:38:29.195 INFO:teuthology.orchestra.run.smithi183.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102-generic amd64 5.15.0-102.112 [2,877 kB] 2024-04-18T21:38:29.254 INFO:teuthology.orchestra.run.smithi183.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.102.99 [2,360 B] 2024-04-18T21:38:29.653 INFO:teuthology.orchestra.run.smithi183.stdout:Fetched 113 MB in 6s (17.5 MB/s) 2024-04-18T21:38:29.754 INFO:teuthology.orchestra.run.smithi097.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-18T21:38:29.759 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../0-linux-modules-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:29.819 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-modules-5.15.0-102-generic. 2024-04-18T21:38:29.900 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.102.99 [1,700 B] 2024-04-18T21:38:29.900 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.102.99 [2,512 B] 2024-04-18T21:38:29.943 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:29.979 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102 all 5.15.0-102.112 [12.4 MB] 2024-04-18T21:38:30.200 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102-generic amd64 5.15.0-102.112 [2,877 kB] 2024-04-18T21:38:30.254 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.102.99 [2,360 B] 2024-04-18T21:38:30.659 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 113 MB in 8s (14.8 MB/s) 2024-04-18T21:38:30.834 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-5.15.0-102-generic. 2024-04-18T21:38:31.797 INFO:teuthology.orchestra.run.smithi183.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-18T21:38:31.801 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../0-linux-modules-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:31.883 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:32.562 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-image-5.15.0-102-generic. 2024-04-18T21:38:32.584 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../1-linux-image-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:32.672 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:32.728 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-18T21:38:32.732 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../0-linux-modules-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:32.831 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:33.190 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-102-generic. 2024-04-18T21:38:33.210 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:33.240 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:34.093 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-image-5.15.0-102-generic. 2024-04-18T21:38:34.114 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../1-linux-image-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:34.199 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:34.671 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-102-generic. 2024-04-18T21:38:34.691 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:34.729 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:34.924 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-image-5.15.0-102-generic. 2024-04-18T21:38:34.945 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../1-linux-image-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:35.018 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:35.494 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-102-generic. 2024-04-18T21:38:35.514 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:35.552 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:39.843 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../3-linux-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:39.985 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:40.334 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:40.472 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-image-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:40.775 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-headers-5.15.0-102. 2024-04-18T21:38:40.797 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../5-linux-headers-5.15.0-102_5.15.0-102.112_all.deb ... 2024-04-18T21:38:40.833 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T21:38:41.419 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../3-linux-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:41.522 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:41.829 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:41.950 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-image-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:41.962 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../3-linux-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:42.087 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:42.262 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-headers-5.15.0-102. 2024-04-18T21:38:42.284 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../5-linux-headers-5.15.0-102_5.15.0-102.112_all.deb ... 2024-04-18T21:38:42.328 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T21:38:42.411 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:42.532 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:42.843 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-headers-5.15.0-102. 2024-04-18T21:38:42.866 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../5-linux-headers-5.15.0-102_5.15.0-102.112_all.deb ... 2024-04-18T21:38:42.909 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T21:38:46.716 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-headers-5.15.0-102-generic. 2024-04-18T21:38:46.753 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../6-linux-headers-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:46.790 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:47.919 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-headers-5.15.0-102-generic. 2024-04-18T21:38:47.938 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../6-linux-headers-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:47.977 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:48.764 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-headers-5.15.0-102-generic. 2024-04-18T21:38:48.801 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../6-linux-headers-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T21:38:48.839 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:49.857 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:49.986 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-headers-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:50.830 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T21:38:51.308 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:51.468 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-headers-generic (5.15.0.102.99) ... 2024-04-18T21:38:51.613 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:51.842 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:51.946 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-headers-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:52.217 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T21:38:52.237 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T21:38:52.340 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T21:38:52.359 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:52.485 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-headers-generic (5.15.0.102.99) ... 2024-04-18T21:38:52.605 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:52.681 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T21:38:52.803 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:52.920 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-generic (5.15.0.102.99) ... 2024-04-18T21:38:52.928 INFO:teuthology.orchestra.run.smithi097.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:38:52.928 INFO:teuthology.orchestra.run.smithi097.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-18T21:38:52.928 INFO:teuthology.orchestra.run.smithi097.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:38:52.929 INFO:teuthology.orchestra.run.smithi097.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-102-generic 2024-04-18T21:38:53.049 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:53.074 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:53.917 INFO:teuthology.orchestra.run.smithi183.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:38:53.917 INFO:teuthology.orchestra.run.smithi183.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-18T21:38:53.917 INFO:teuthology.orchestra.run.smithi183.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:38:53.917 INFO:teuthology.orchestra.run.smithi183.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-102-generic 2024-04-18T21:38:54.074 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:54.371 INFO:teuthology.orchestra.run.smithi104.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:38:54.371 INFO:teuthology.orchestra.run.smithi104.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-18T21:38:54.372 INFO:teuthology.orchestra.run.smithi104.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:38:54.372 INFO:teuthology.orchestra.run.smithi104.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-102-generic 2024-04-18T21:38:54.475 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:54.527 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:55.458 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:55.877 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-image-generic (5.15.0.102.99) ... 2024-04-18T21:38:55.928 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:55.986 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-generic (5.15.0.102.99) ... 2024-04-18T21:38:56.112 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:56.169 INFO:teuthology.orchestra.run.smithi097.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-18T21:38:56.170 INFO:teuthology.orchestra.run.smithi097.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:38:57.194 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-image-generic (5.15.0.102.99) ... 2024-04-18T21:38:57.319 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-generic (5.15.0.102.99) ... 2024-04-18T21:38:57.363 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-generic (5.15.0.102.99) ... 2024-04-18T21:38:57.437 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:57.489 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-generic (5.15.0.102.99) ... 2024-04-18T21:38:57.502 INFO:teuthology.orchestra.run.smithi183.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-18T21:38:57.502 INFO:teuthology.orchestra.run.smithi183.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:38:57.615 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T21:38:57.683 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-18T21:38:57.683 INFO:teuthology.orchestra.run.smithi104.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:39:10.821 INFO:teuthology.orchestra.run.smithi097.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-18T21:39:10.822 INFO:teuthology.orchestra.run.smithi097.stdout:Sourcing file `/etc/default/grub' 2024-04-18T21:39:10.845 INFO:teuthology.orchestra.run.smithi097.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T21:39:10.859 INFO:teuthology.orchestra.run.smithi097.stdout:Generating grub configuration file ... 2024-04-18T21:39:11.159 INFO:teuthology.orchestra.run.smithi097.stdout:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:39:11.182 INFO:teuthology.orchestra.run.smithi097.stdout:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:39:11.422 INFO:teuthology.orchestra.run.smithi183.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-18T21:39:11.423 INFO:teuthology.orchestra.run.smithi183.stdout:Sourcing file `/etc/default/grub' 2024-04-18T21:39:11.434 INFO:teuthology.orchestra.run.smithi183.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T21:39:11.452 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-18T21:39:11.452 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub' 2024-04-18T21:39:11.460 INFO:teuthology.orchestra.run.smithi183.stdout:Generating grub configuration file ... 2024-04-18T21:39:11.463 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T21:39:11.486 INFO:teuthology.orchestra.run.smithi104.stdout:Generating grub configuration file ... 2024-04-18T21:39:11.515 INFO:teuthology.orchestra.run.smithi097.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:39:11.521 INFO:teuthology.orchestra.run.smithi097.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T21:39:11.592 INFO:teuthology.orchestra.run.smithi097.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T21:39:11.598 INFO:teuthology.orchestra.run.smithi097.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T21:39:11.746 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:39:11.756 INFO:teuthology.orchestra.run.smithi097.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T21:39:11.756 INFO:teuthology.orchestra.run.smithi097.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T21:39:11.756 INFO:teuthology.orchestra.run.smithi097.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T21:39:11.764 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:39:11.796 INFO:teuthology.orchestra.run.smithi097.stdout:done 2024-04-18T21:39:11.800 INFO:teuthology.orchestra.run.smithi183.stdout:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:39:11.820 INFO:teuthology.orchestra.run.smithi183.stdout:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:39:12.065 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:39:12.070 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T21:39:12.146 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T21:39:12.151 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T21:39:12.174 INFO:teuthology.orchestra.run.smithi183.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:39:12.180 INFO:teuthology.orchestra.run.smithi183.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T21:39:12.247 INFO:teuthology.orchestra.run.smithi183.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T21:39:12.252 INFO:teuthology.orchestra.run.smithi183.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T21:39:12.274 DEBUG:teuthology.orchestra.run.smithi097:> dpkg -s linux-image-generic 2024-04-18T21:39:12.318 INFO:teuthology.orchestra.run.smithi104.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T21:39:12.318 INFO:teuthology.orchestra.run.smithi104.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T21:39:12.318 INFO:teuthology.orchestra.run.smithi104.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T21:39:12.338 INFO:teuthology.orchestra.run.smithi097.stdout:Package: linux-image-generic 2024-04-18T21:39:12.338 INFO:teuthology.orchestra.run.smithi097.stdout:Status: install ok installed 2024-04-18T21:39:12.338 INFO:teuthology.orchestra.run.smithi097.stdout:Priority: optional 2024-04-18T21:39:12.338 INFO:teuthology.orchestra.run.smithi097.stdout:Section: kernel 2024-04-18T21:39:12.338 INFO:teuthology.orchestra.run.smithi097.stdout:Installed-Size: 21 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: amd64 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout:Source: linux-meta 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout:Version: 5.15.0.102.99 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout:Recommends: thermald 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout:Description: Generic Linux kernel image 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout: This package will always depend on the latest generic kernel image 2024-04-18T21:39:12.339 INFO:teuthology.orchestra.run.smithi097.stdout: available. 2024-04-18T21:39:12.339 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-102-generic 2024-04-18T21:39:12.340 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-18T21:39:12.340 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-18T21:39:12.340 INFO:teuthology.task.kernel:Installing distro kernel on osd.0... 2024-04-18T21:39:12.341 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2024-04-18T21:39:12.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get clean 2024-04-18T21:39:12.356 INFO:teuthology.orchestra.run.smithi104.stdout:done 2024-04-18T21:39:12.436 INFO:teuthology.orchestra.run.smithi183.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T21:39:12.436 INFO:teuthology.orchestra.run.smithi183.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T21:39:12.436 INFO:teuthology.orchestra.run.smithi183.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T21:39:12.481 INFO:teuthology.orchestra.run.smithi183.stdout:done 2024-04-18T21:39:12.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get update 2024-04-18T21:39:12.654 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:39:12.665 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T21:39:12.685 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T21:39:12.768 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T21:39:12.792 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic 2024-04-18T21:39:12.816 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 21 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.15.0.102.99 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2024-04-18T21:39:12.817 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2024-04-18T21:39:12.818 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2024-04-18T21:39:12.818 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-102-generic 2024-04-18T21:39:12.818 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-18T21:39:12.818 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-18T21:39:12.818 INFO:teuthology.task.kernel:Installing distro kernel on osd.4... 2024-04-18T21:39:12.818 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-04-18T21:39:12.818 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2024-04-18T21:39:12.939 DEBUG:teuthology.orchestra.run.smithi183:> dpkg -s linux-image-generic 2024-04-18T21:39:12.961 INFO:teuthology.orchestra.run.smithi183.stdout:Package: linux-image-generic 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Status: install ok installed 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Priority: optional 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Section: kernel 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Installed-Size: 21 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: amd64 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Source: linux-meta 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Version: 5.15.0.102.99 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Recommends: thermald 2024-04-18T21:39:12.962 INFO:teuthology.orchestra.run.smithi183.stdout:Description: Generic Linux kernel image 2024-04-18T21:39:12.963 INFO:teuthology.orchestra.run.smithi183.stdout: This package will always depend on the latest generic kernel image 2024-04-18T21:39:12.963 INFO:teuthology.orchestra.run.smithi183.stdout: available. 2024-04-18T21:39:12.963 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-102-generic 2024-04-18T21:39:12.963 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-18T21:39:12.963 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-18T21:39:12.963 INFO:teuthology.task.kernel:Installing distro kernel on osd.8... 2024-04-18T21:39:12.963 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-04-18T21:39:12.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo apt-get clean 2024-04-18T21:39:13.008 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-04-18T21:39:13.145 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T21:39:13.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo apt-get update 2024-04-18T21:39:13.261 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:39:13.280 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T21:39:13.354 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T21:39:13.395 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:39:13.425 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T21:39:13.477 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T21:39:13.560 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T21:39:14.055 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2024-04-18T21:39:14.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T21:39:14.124 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2024-04-18T21:39:14.302 INFO:teuthology.orchestra.run.smithi097.stdout:Building dependency tree... 2024-04-18T21:39:14.303 INFO:teuthology.orchestra.run.smithi097.stdout:Reading state information... 2024-04-18T21:39:14.463 INFO:teuthology.orchestra.run.smithi097.stdout:linux-image-generic is already the newest version (5.15.0.102.99). 2024-04-18T21:39:14.463 INFO:teuthology.orchestra.run.smithi097.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:39:14.463 INFO:teuthology.orchestra.run.smithi097.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:39:14.463 INFO:teuthology.orchestra.run.smithi097.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:39:14.463 INFO:teuthology.orchestra.run.smithi097.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:39:14.464 INFO:teuthology.orchestra.run.smithi097.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T21:39:14.464 INFO:teuthology.orchestra.run.smithi097.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T21:39:14.464 INFO:teuthology.orchestra.run.smithi097.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:39:14.503 INFO:teuthology.orchestra.run.smithi097.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T21:39:14.504 DEBUG:teuthology.orchestra.run.smithi097:> dpkg -s linux-image-generic 2024-04-18T21:39:14.553 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-04-18T21:39:14.557 INFO:teuthology.orchestra.run.smithi097.stdout:Package: linux-image-generic 2024-04-18T21:39:14.557 INFO:teuthology.orchestra.run.smithi097.stdout:Status: install ok installed 2024-04-18T21:39:14.557 INFO:teuthology.orchestra.run.smithi097.stdout:Priority: optional 2024-04-18T21:39:14.557 INFO:teuthology.orchestra.run.smithi097.stdout:Section: kernel 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Installed-Size: 21 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: amd64 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Source: linux-meta 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Version: 5.15.0.102.99 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Recommends: thermald 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout:Description: Generic Linux kernel image 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout: This package will always depend on the latest generic kernel image 2024-04-18T21:39:14.558 INFO:teuthology.orchestra.run.smithi097.stdout: available. 2024-04-18T21:39:14.559 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-04-18T21:39:14.571 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T21:39:14.602 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.vTGn0bcTJ6 2024-04-18T21:39:14.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vTGn0bcTJ6 2024-04-18T21:39:14.632 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-04-18T21:39:14.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.vTGn0bcTJ6 2024-04-18T21:39:14.707 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2024-04-18T21:39:14.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T21:39:14.775 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.vTGn0bcTJ6 is 10KB 2024-04-18T21:39:14.775 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2024-04-18T21:39:14.786 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.vTGn0bcTJ6 2024-04-18T21:39:14.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-18T21:39:14.833 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-04-18T21:39:14.833 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-04-18T21:39:14.845 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:39:14.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-18T21:39:14.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-18T21:39:14.924 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-102-generic 2024-04-18T21:39:14.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo update-grub 2024-04-18T21:39:14.960 INFO:teuthology.orchestra.run.smithi183.stdout:Building dependency tree... 2024-04-18T21:39:14.961 INFO:teuthology.orchestra.run.smithi183.stdout:Reading state information... 2024-04-18T21:39:15.008 INFO:teuthology.orchestra.run.smithi104.stdout:linux-image-generic is already the newest version (5.15.0.102.99). 2024-04-18T21:39:15.008 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:39:15.009 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:39:15.009 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:39:15.009 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:39:15.009 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T21:39:15.009 INFO:teuthology.orchestra.run.smithi104.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T21:39:15.009 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:39:15.051 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T21:39:15.053 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic 2024-04-18T21:39:15.069 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic 2024-04-18T21:39:15.069 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2024-04-18T21:39:15.069 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2024-04-18T21:39:15.069 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2024-04-18T21:39:15.069 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 21 2024-04-18T21:39:15.069 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T21:39:15.069 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2024-04-18T21:39:15.070 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta 2024-04-18T21:39:15.070 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.15.0.102.99 2024-04-18T21:39:15.070 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T21:39:15.070 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T21:39:15.070 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2024-04-18T21:39:15.070 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2024-04-18T21:39:15.070 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2024-04-18T21:39:15.070 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2024-04-18T21:39:15.070 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-04-18T21:39:15.116 INFO:teuthology.orchestra.run.smithi183.stdout:linux-image-generic is already the newest version (5.15.0.102.99). 2024-04-18T21:39:15.116 INFO:teuthology.orchestra.run.smithi183.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:39:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:39:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:39:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:39:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T21:39:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T21:39:15.117 INFO:teuthology.orchestra.run.smithi183.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:39:15.121 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.4DN7eJVmT8 2024-04-18T21:39:15.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub/grub.cfg /tmp/tmp.4DN7eJVmT8 2024-04-18T21:39:15.155 INFO:teuthology.orchestra.run.smithi183.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T21:39:15.156 DEBUG:teuthology.orchestra.run.smithi183:> dpkg -s linux-image-generic 2024-04-18T21:39:15.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.4DN7eJVmT8 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Package: linux-image-generic 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Status: install ok installed 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Priority: optional 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Section: kernel 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Installed-Size: 21 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: amd64 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Source: linux-meta 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Version: 5.15.0.102.99 2024-04-18T21:39:15.212 INFO:teuthology.orchestra.run.smithi183.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T21:39:15.213 INFO:teuthology.orchestra.run.smithi183.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T21:39:15.213 INFO:teuthology.orchestra.run.smithi183.stdout:Recommends: thermald 2024-04-18T21:39:15.213 INFO:teuthology.orchestra.run.smithi183.stdout:Description: Generic Linux kernel image 2024-04-18T21:39:15.213 INFO:teuthology.orchestra.run.smithi183.stdout: This package will always depend on the latest generic kernel image 2024-04-18T21:39:15.213 INFO:teuthology.orchestra.run.smithi183.stdout: available. 2024-04-18T21:39:15.213 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-04-18T21:39:15.257 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.H5kFzKmR6D 2024-04-18T21:39:15.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub/grub.cfg /tmp/tmp.H5kFzKmR6D 2024-04-18T21:39:15.300 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.4DN7eJVmT8 is 10KB 2024-04-18T21:39:15.313 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.4DN7eJVmT8 2024-04-18T21:39:15.314 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.H5kFzKmR6D 2024-04-18T21:39:15.319 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-18T21:39:15.373 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:39:15.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-18T21:39:15.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-18T21:39:15.433 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.H5kFzKmR6D is 10KB 2024-04-18T21:39:15.438 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-102-generic 2024-04-18T21:39:15.438 DEBUG:teuthology.orchestra.run.smithi104:> sudo update-grub 2024-04-18T21:39:15.488 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.H5kFzKmR6D 2024-04-18T21:39:15.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-18T21:39:15.549 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:39:15.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-18T21:39:15.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-18T21:39:15.617 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-102-generic 2024-04-18T21:39:15.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo update-grub 2024-04-18T21:39:16.153 INFO:teuthology.orchestra.run.smithi097.stderr:Sourcing file `/etc/default/grub' 2024-04-18T21:39:16.154 INFO:teuthology.orchestra.run.smithi097.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T21:39:16.163 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-04-18T21:39:16.436 INFO:teuthology.orchestra.run.smithi097.stderr:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:39:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:39:16.611 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub' 2024-04-18T21:39:16.613 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T21:39:16.615 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-04-18T21:39:16.730 INFO:teuthology.orchestra.run.smithi097.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:39:16.736 INFO:teuthology.orchestra.run.smithi097.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T21:39:16.775 INFO:teuthology.orchestra.run.smithi097.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T21:39:16.780 INFO:teuthology.orchestra.run.smithi097.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T21:39:16.805 INFO:teuthology.orchestra.run.smithi183.stderr:Sourcing file `/etc/default/grub' 2024-04-18T21:39:16.806 INFO:teuthology.orchestra.run.smithi183.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T21:39:16.809 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-04-18T21:39:16.810 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:39:16.816 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:39:16.933 INFO:teuthology.orchestra.run.smithi097.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T21:39:16.933 INFO:teuthology.orchestra.run.smithi097.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T21:39:16.933 INFO:teuthology.orchestra.run.smithi097.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T21:39:16.944 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-04-18T21:39:16.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-04-18T21:39:17.053 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:39:17.055 INFO:teuthology.orchestra.run.smithi183.stderr:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T21:39:17.059 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T21:39:17.061 INFO:teuthology.orchestra.run.smithi183.stderr:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T21:39:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T21:39:17.107 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T21:39:17.226 INFO:teuthology.orchestra.run.smithi104.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T21:39:17.226 INFO:teuthology.orchestra.run.smithi104.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T21:39:17.226 INFO:teuthology.orchestra.run.smithi104.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T21:39:17.251 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-04-18T21:39:17.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-04-18T21:39:17.332 INFO:teuthology.orchestra.run.smithi183.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T21:39:17.338 INFO:teuthology.orchestra.run.smithi183.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T21:39:17.376 INFO:teuthology.orchestra.run.smithi183.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T21:39:17.381 INFO:teuthology.orchestra.run.smithi183.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T21:39:17.496 INFO:teuthology.orchestra.run.smithi183.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T21:39:17.496 INFO:teuthology.orchestra.run.smithi183.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T21:39:17.497 INFO:teuthology.orchestra.run.smithi183.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T21:39:17.521 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-04-18T21:39:17.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-04-18T21:39:46.978 INFO:teuthology.task.kernel:Checking client osd.0 for new kernel version... 2024-04-18T21:39:46.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-18T21:39:46.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:39:47.264 INFO:teuthology.task.kernel:Checking client osd.4 for new kernel version... 2024-04-18T21:39:47.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-18T21:39:47.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:39:47.533 INFO:teuthology.task.kernel:Checking client osd.8 for new kernel version... 2024-04-18T21:39:47.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-18T21:39:47.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:40:05.539 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-04-18T21:40:05.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-04-18T21:40:06.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-04-18T21:40:14.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-18T21:40:14.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:40:14.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-18T21:40:14.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:40:15.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-18T21:40:15.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:40:17.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-04-18T21:40:17.856 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-04-18T21:40:18.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-04-18T21:40:29.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-18T21:40:29.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:40:29.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-18T21:40:29.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:40:30.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-18T21:40:30.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:40:48.031 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-04-18T21:40:48.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-04-18T21:40:48.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-04-18T21:41:03.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-18T21:41:03.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:41:03.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-18T21:41:03.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:41:03.316 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-04-18T21:41:03.521 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-04-18T21:41:03.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-18T21:41:03.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:41:03.848 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-04-18T21:41:03.910 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-18T21:41:03.911 INFO:teuthology.task.kernel:Checking kernel version of osd.0, want "5.15.0-102-generic"... 2024-04-18T21:41:03.911 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-04-18T21:41:03.959 INFO:teuthology.orchestra.run.smithi097.stdout:5.15.0-102-generic 2024-04-18T21:41:03.959 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-102-generic vs 5.15.0-102-generic 2024-04-18T21:41:03.959 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-18T21:41:03.959 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-18T21:41:04.142 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-18T21:41:04.142 INFO:teuthology.task.kernel:Checking kernel version of osd.4, want "5.15.0-102-generic"... 2024-04-18T21:41:04.142 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-04-18T21:41:04.194 INFO:teuthology.orchestra.run.smithi104.stdout:5.15.0-102-generic 2024-04-18T21:41:04.195 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-102-generic vs 5.15.0-102-generic 2024-04-18T21:41:04.195 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-18T21:41:04.195 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-18T21:41:04.424 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-18T21:41:04.425 INFO:teuthology.task.kernel:Checking kernel version of osd.8, want "5.15.0-102-generic"... 2024-04-18T21:41:04.425 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-04-18T21:41:04.470 INFO:teuthology.orchestra.run.smithi183.stdout:5.15.0-102-generic 2024-04-18T21:41:04.470 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-102-generic vs 5.15.0-102-generic 2024-04-18T21:41:04.470 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-18T21:41:04.470 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-18T21:41:04.960 INFO:teuthology.task.kernel:Enabling kdb on osd.0... 2024-04-18T21:41:04.960 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-18T21:41:05.028 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2024-04-18T21:41:05.069 DEBUG:teuthology.parallel:result is None 2024-04-18T21:41:05.196 INFO:teuthology.task.kernel:Enabling kdb on osd.4... 2024-04-18T21:41:05.196 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-18T21:41:05.471 INFO:teuthology.task.kernel:Enabling kdb on osd.8... 2024-04-18T21:41:05.472 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-18T21:41:05.521 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-04-18T21:41:05.529 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-04-18T21:41:05.544 DEBUG:teuthology.parallel:result is None 2024-04-18T21:41:05.567 DEBUG:teuthology.parallel:result is None 2024-04-18T21:41:05.568 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-18T21:41:05.578 INFO:teuthology.task.internal:Creating test directory... 2024-04-18T21:41:05.578 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-18T21:41:05.581 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-18T21:41:05.584 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-18T21:41:05.590 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-18T21:41:05.617 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-18T21:41:05.686 INFO:teuthology.task.internal:Creating archive directory... 2024-04-18T21:41:05.687 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-18T21:41:05.690 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-18T21:41:05.693 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-18T21:41:05.721 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-18T21:41:05.728 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-18T21:41:05.728 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-18T21:41:05.752 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-18T21:41:05.756 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-18T21:41:05.771 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T21:41:05.778 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T21:41:05.779 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T21:41:05.780 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T21:41:05.786 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T21:41:05.788 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T21:41:05.790 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-18T21:41:05.798 INFO:teuthology.task.internal:Configuring sudo... 2024-04-18T21:41:05.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-18T21:41:05.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-18T21:41:05.835 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-18T21:41:05.848 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-18T21:41:05.860 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-18T21:41:05.860 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-18T21:41:05.883 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-18T21:41:05.891 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-18T21:41:05.897 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-18T21:41:05.932 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-18T21:41:05.980 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:41:05.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-18T21:41:06.045 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-18T21:41:06.052 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-18T21:41:06.099 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:41:06.099 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-18T21:41:06.166 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-18T21:41:06.172 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-18T21:41:06.218 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:41:06.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-18T21:41:06.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-04-18T21:41:06.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-04-18T21:41:06.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-04-18T21:41:06.361 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-18T21:41:06.373 INFO:teuthology.task.internal:Starting timer... 2024-04-18T21:41:06.373 INFO:teuthology.run_tasks:Running task pcp... 2024-04-18T21:41:06.398 INFO:teuthology.run_tasks:Running task selinux... 2024-04-18T21:41:06.432 DEBUG:teuthology.task.selinux:Excluding smithi097: OS 'ubuntu' does not support SELinux 2024-04-18T21:41:06.432 DEBUG:teuthology.task.selinux:Excluding smithi104: OS 'ubuntu' does not support SELinux 2024-04-18T21:41:06.432 DEBUG:teuthology.task.selinux:Excluding smithi183: OS 'ubuntu' does not support SELinux 2024-04-18T21:41:06.432 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-18T21:41:06.432 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-18T21:41:06.433 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-18T21:41:06.433 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-18T21:41:06.469 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-18T21:41:06.488 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-18T21:41:06.489 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi104.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-18T21:48:21.149 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-04-18T21:48:21.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-18T21:48:21.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:48:21.231 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-04-18T21:48:21.306 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-18T21:48:21.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-18T21:48:21.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:48:21.387 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-04-18T21:48:21.462 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-18T21:48:21.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-18T21:48:21.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T21:48:21.542 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-04-18T21:48:21.619 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-18T21:48:21.619 INFO:teuthology.run_tasks:Running task clock... 2024-04-18T21:48:21.629 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-18T21:48:21.629 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-18T21:48:21.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T21:48:21.632 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-18T21:48:21.632 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T21:48:21.634 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-18T21:48:21.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T21:48:21.659 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-18T21:48:21.659 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Command line: ntpd -gq 2024-04-18T21:48:21.659 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: ---------------------------------------------------- 2024-04-18T21:48:21.659 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: ntp-4 is maintained by Network Time Foundation, 2024-04-18T21:48:21.660 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-18T21:48:21.660 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: corporation. Support and training for ntp-4 are 2024-04-18T21:48:21.660 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: available at https://www.nwtime.org/support 2024-04-18T21:48:21.660 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: ---------------------------------------------------- 2024-04-18T21:48:21.660 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: proto: precision = 0.060 usec (-24) 2024-04-18T21:48:21.661 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: basedate set to 2022-02-04 2024-04-18T21:48:21.661 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: gps base set to 2022-02-06 (week 2196) 2024-04-18T21:48:21.661 INFO:teuthology.orchestra.run.smithi097.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T21:48:21.661 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T21:48:21.662 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: restrict ::: KOD does nothing without LIMITED. 2024-04-18T21:48:21.662 INFO:teuthology.orchestra.run.smithi097.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-18T21:48:21.663 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Listen and drop on 0 v6wildcard [::]:123 2024-04-18T21:48:21.663 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-18T21:48:21.663 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Listen normally on 2 lo 127.0.0.1:123 2024-04-18T21:48:21.663 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Listen normally on 3 enp3s0f1 172.21.15.97:123 2024-04-18T21:48:21.663 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Listen normally on 4 lo [::1]:123 2024-04-18T21:48:21.663 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d805%5]:123 2024-04-18T21:48:21.663 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:21 ntpd[17241]: Listening on routing socket on fd #22 for interface updates 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Command line: ntpd -gq 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: ---------------------------------------------------- 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: ntp-4 is maintained by Network Time Foundation, 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: corporation. Support and training for ntp-4 are 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: available at https://www.nwtime.org/support 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: ---------------------------------------------------- 2024-04-18T21:48:21.666 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: proto: precision = 0.058 usec (-24) 2024-04-18T21:48:21.667 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: basedate set to 2022-02-04 2024-04-18T21:48:21.667 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: gps base set to 2022-02-06 (week 2196) 2024-04-18T21:48:21.667 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T21:48:21.667 INFO:teuthology.orchestra.run.smithi104.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T21:48:21.667 INFO:teuthology.orchestra.run.smithi104.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-18T21:48:21.668 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: restrict ::: KOD does nothing without LIMITED. 2024-04-18T21:48:21.668 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Listen and drop on 0 v6wildcard [::]:123 2024-04-18T21:48:21.668 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-18T21:48:21.668 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Listen normally on 2 lo 127.0.0.1:123 2024-04-18T21:48:21.668 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Listen normally on 3 enp3s0f1 172.21.15.104:123 2024-04-18T21:48:21.668 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Listen normally on 4 lo [::1]:123 2024-04-18T21:48:21.669 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce89%5]:123 2024-04-18T21:48:21.669 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:21 ntpd[17198]: Listening on routing socket on fd #22 for interface updates 2024-04-18T21:48:21.688 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-18T21:48:21.688 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Command line: ntpd -gq 2024-04-18T21:48:21.689 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: ---------------------------------------------------- 2024-04-18T21:48:21.689 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: ntp-4 is maintained by Network Time Foundation, 2024-04-18T21:48:21.689 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-18T21:48:21.689 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: corporation. Support and training for ntp-4 are 2024-04-18T21:48:21.689 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: available at https://www.nwtime.org/support 2024-04-18T21:48:21.689 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: ---------------------------------------------------- 2024-04-18T21:48:21.690 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: proto: precision = 0.043 usec (-24) 2024-04-18T21:48:21.690 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: basedate set to 2022-02-04 2024-04-18T21:48:21.690 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: gps base set to 2022-02-06 (week 2196) 2024-04-18T21:48:21.690 INFO:teuthology.orchestra.run.smithi183.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T21:48:21.691 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T21:48:21.691 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: restrict ::: KOD does nothing without LIMITED. 2024-04-18T21:48:21.691 INFO:teuthology.orchestra.run.smithi183.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-18T21:48:21.692 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Listen and drop on 0 v6wildcard [::]:123 2024-04-18T21:48:21.692 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-18T21:48:21.692 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Listen normally on 2 lo 127.0.0.1:123 2024-04-18T21:48:21.692 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Listen normally on 3 enp3s0f1 172.21.15.183:123 2024-04-18T21:48:21.692 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Listen normally on 4 lo [::1]:123 2024-04-18T21:48:21.692 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8085%5]:123 2024-04-18T21:48:21.693 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:21 ntpd[17178]: Listening on routing socket on fd #22 for interface updates 2024-04-18T21:48:22.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:22 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:22.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:22 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:22.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:22 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:22.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:22 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:22.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:22 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:22.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:22 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:22 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:22 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:22 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:22 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:22 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:22 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:22 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:22 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:23.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:23 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:23.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:23 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:23.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:23 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:23.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:23 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:23.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:23 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:23.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:23 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:23.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:23.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:23.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:23 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:23.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:23 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:23.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:23 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:23.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:23 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:23.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:23 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:23.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:23 ntpd[17178]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:23.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:23 ntpd[17178]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:24 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:24 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:24 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:24 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:24 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:24 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:24.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:24 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:24.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:24 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:24.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:24 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:24.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:24 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:24.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:24 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:24.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:24 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:24.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:24 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:24.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:24 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:25 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:25 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:25 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:25 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:25 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:25 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:25 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:25.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:25 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:25.668 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:25 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:25.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:25.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:25 ntpd[17178]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:26.662 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:26 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:26.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:26 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:26.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:26 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:26.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:26 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:26.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:26 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:26.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:26 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:26.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:26 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:26.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:26 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:26.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:26 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:26.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:26 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:26.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:26 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:26.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:26 ntpd[17178]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:26.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:26 ntpd[17178]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:27.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:27 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:27 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:27 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.660 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:27 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:27 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:27.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:27 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:27.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:27.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:27 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:27.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:27.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:27 ntpd[17178]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:28.661 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:28.662 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:28.662 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:28.662 INFO:teuthology.orchestra.run.smithi097.stdout:18 Apr 21:48:28 ntpd[17241]: ntpd: time slew -0.002325 s 2024-04-18T21:48:28.662 INFO:teuthology.orchestra.run.smithi097.stdout:ntpd: time slew -0.002325s 2024-04-18T21:48:28.662 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-18T21:48:28.663 INFO:teuthology.orchestra.run.smithi097.stderr:18 Apr 21:48:28 ntpd[17241]: can't open /var/log/ntpstats/loopstats.20240418: Permission denied 2024-04-18T21:48:28.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:28 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:28.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:28 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:28.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:28 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:28.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:28 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:28.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:28 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:28.666 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:28 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:28.667 INFO:teuthology.orchestra.run.smithi104.stdout:18 Apr 21:48:28 ntpd[17198]: ntpd: time slew -0.000421 s 2024-04-18T21:48:28.667 INFO:teuthology.orchestra.run.smithi104.stdout:ntpd: time slew -0.000421s 2024-04-18T21:48:28.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:28 ntpd[17198]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-18T21:48:28.667 INFO:teuthology.orchestra.run.smithi104.stderr:18 Apr 21:48:28 ntpd[17198]: can't open /var/log/ntpstats/loopstats.20240418: Permission denied 2024-04-18T21:48:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:28 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:28 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:28 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:28 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:28 ntpd[17178]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:28 ntpd[17178]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T21:48:28.715 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2024-04-18T21:48:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:28.717 INFO:teuthology.orchestra.run.smithi097.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T21:48:28.717 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================== 2024-04-18T21:48:28.717 INFO:teuthology.orchestra.run.smithi097.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:28.717 INFO:teuthology.orchestra.run.smithi097.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:28.717 INFO:teuthology.orchestra.run.smithi097.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:28.717 INFO:teuthology.orchestra.run.smithi097.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:29.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:29.689 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:29.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:29.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:29.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T21:48:29.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T21:48:29.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T21:48:29.690 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T21:48:29.690 INFO:teuthology.orchestra.run.smithi183.stdout:18 Apr 21:48:29 ntpd[17178]: ntpd: time slew -0.001407 s 2024-04-18T21:48:29.690 INFO:teuthology.orchestra.run.smithi183.stdout:ntpd: time slew -0.001407s 2024-04-18T21:48:29.691 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-18T21:48:29.691 INFO:teuthology.orchestra.run.smithi183.stderr:18 Apr 21:48:29 ntpd[17178]: can't open /var/log/ntpstats/loopstats.20240418: Permission denied 2024-04-18T21:48:30.051 INFO:teuthology.orchestra.run.smithi183.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T21:48:30.051 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================== 2024-04-18T21:48:30.051 INFO:teuthology.orchestra.run.smithi183.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:30.051 INFO:teuthology.orchestra.run.smithi183.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:30.052 INFO:teuthology.orchestra.run.smithi183.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:30.052 INFO:teuthology.orchestra.run.smithi183.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:48:30.053 INFO:teuthology.run_tasks:Running task install... 2024-04-18T21:48:30.061 DEBUG:teuthology.task.install:project ceph 2024-04-18T21:48:30.062 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}} 2024-04-18T21:48:30.062 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'} 2024-04-18T21:48:30.062 INFO:teuthology.task.install:Using flavor: default 2024-04-18T21:48:30.068 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-18T21:48:30.068 INFO:teuthology.task.install:extra packages: [] 2024-04-18T21:48:30.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-key list | grep Ceph 2024-04-18T21:48:30.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-key list | grep Ceph 2024-04-18T21:48:30.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo apt-key list | grep Ceph 2024-04-18T21:48:30.131 INFO:teuthology.orchestra.run.smithi097.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-18T21:48:30.132 INFO:teuthology.orchestra.run.smithi104.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-18T21:48:30.160 INFO:teuthology.orchestra.run.smithi097.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-18T21:48:30.160 INFO:teuthology.orchestra.run.smithi097.stdout:uid [ unknown] Ceph.com (release key) 2024-04-18T21:48:30.160 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-18T21:48:30.161 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-18T21:48:30.165 INFO:teuthology.orchestra.run.smithi104.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-18T21:48:30.165 INFO:teuthology.orchestra.run.smithi104.stdout:uid [ unknown] Ceph.com (release key) 2024-04-18T21:48:30.166 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-18T21:48:30.166 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-18T21:48:30.169 INFO:teuthology.orchestra.run.smithi183.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-18T21:48:30.196 INFO:teuthology.orchestra.run.smithi183.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-18T21:48:30.196 INFO:teuthology.orchestra.run.smithi183.stdout:uid [ unknown] Ceph.com (release key) 2024-04-18T21:48:30.197 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-18T21:48:30.197 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-18T21:48:30.337 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default/ 2024-04-18T21:48:30.337 INFO:teuthology.task.install.deb:Package version is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:48:30.379 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default/ 2024-04-18T21:48:30.379 INFO:teuthology.task.install.deb:Package version is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:48:30.383 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default/ 2024-04-18T21:48:30.383 INFO:teuthology.task.install.deb:Package version is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:48:30.615 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:48:30.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-18T21:48:30.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo apt-get update 2024-04-18T21:48:30.656 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:48:30.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-18T21:48:30.664 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:48:30.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-18T21:48:30.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get update 2024-04-18T21:48:30.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-04-18T21:48:30.798 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:48:30.798 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T21:48:30.814 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T21:48:30.835 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T21:48:30.847 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:48:30.857 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T21:48:30.877 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T21:48:30.960 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T21:48:30.974 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T21:48:30.974 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T21:48:30.977 INFO:teuthology.orchestra.run.smithi183.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy InRelease 2024-04-18T21:48:31.034 INFO:teuthology.orchestra.run.smithi183.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-18T21:48:31.035 INFO:teuthology.orchestra.run.smithi104.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy InRelease 2024-04-18T21:48:31.046 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T21:48:31.091 INFO:teuthology.orchestra.run.smithi097.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy InRelease 2024-04-18T21:48:31.092 INFO:teuthology.orchestra.run.smithi183.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-18T21:48:31.093 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-18T21:48:31.129 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T21:48:31.151 INFO:teuthology.orchestra.run.smithi183.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,425 B] 2024-04-18T21:48:31.153 INFO:teuthology.orchestra.run.smithi104.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-18T21:48:31.165 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-18T21:48:31.213 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-18T21:48:31.219 INFO:teuthology.orchestra.run.smithi183.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-18T21:48:31.243 INFO:teuthology.orchestra.run.smithi097.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-18T21:48:31.307 INFO:teuthology.orchestra.run.smithi183.stdout:Fetched 29.0 kB in 0s (59.2 kB/s) 2024-04-18T21:48:31.321 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-18T21:48:31.341 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,425 B] 2024-04-18T21:48:31.410 INFO:teuthology.orchestra.run.smithi097.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,425 B] 2024-04-18T21:48:31.424 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 29.0 kB in 1s (51.9 kB/s) 2024-04-18T21:48:31.504 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 29.0 kB in 1s (45.8 kB/s) 2024-04-18T21:48:32.299 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2024-04-18T21:48:32.314 DEBUG:teuthology.orchestra.run.smithi183:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2369-g1a6a3e74-1jammy cephadm=19.0.0-2369-g1a6a3e74-1jammy ceph-mds=19.0.0-2369-g1a6a3e74-1jammy ceph-mgr=19.0.0-2369-g1a6a3e74-1jammy ceph-common=19.0.0-2369-g1a6a3e74-1jammy ceph-fuse=19.0.0-2369-g1a6a3e74-1jammy ceph-test=19.0.0-2369-g1a6a3e74-1jammy ceph-volume=19.0.0-2369-g1a6a3e74-1jammy radosgw=19.0.0-2369-g1a6a3e74-1jammy python3-rados=19.0.0-2369-g1a6a3e74-1jammy python3-rgw=19.0.0-2369-g1a6a3e74-1jammy python3-cephfs=19.0.0-2369-g1a6a3e74-1jammy python3-rbd=19.0.0-2369-g1a6a3e74-1jammy libcephfs2=19.0.0-2369-g1a6a3e74-1jammy libcephfs-dev=19.0.0-2369-g1a6a3e74-1jammy librados2=19.0.0-2369-g1a6a3e74-1jammy librbd1=19.0.0-2369-g1a6a3e74-1jammy rbd-fuse=19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:48:32.376 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2024-04-18T21:48:32.415 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-04-18T21:48:32.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2369-g1a6a3e74-1jammy cephadm=19.0.0-2369-g1a6a3e74-1jammy ceph-mds=19.0.0-2369-g1a6a3e74-1jammy ceph-mgr=19.0.0-2369-g1a6a3e74-1jammy ceph-common=19.0.0-2369-g1a6a3e74-1jammy ceph-fuse=19.0.0-2369-g1a6a3e74-1jammy ceph-test=19.0.0-2369-g1a6a3e74-1jammy ceph-volume=19.0.0-2369-g1a6a3e74-1jammy radosgw=19.0.0-2369-g1a6a3e74-1jammy python3-rados=19.0.0-2369-g1a6a3e74-1jammy python3-rgw=19.0.0-2369-g1a6a3e74-1jammy python3-cephfs=19.0.0-2369-g1a6a3e74-1jammy python3-rbd=19.0.0-2369-g1a6a3e74-1jammy libcephfs2=19.0.0-2369-g1a6a3e74-1jammy libcephfs-dev=19.0.0-2369-g1a6a3e74-1jammy librados2=19.0.0-2369-g1a6a3e74-1jammy librbd1=19.0.0-2369-g1a6a3e74-1jammy rbd-fuse=19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:48:32.474 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2024-04-18T21:48:32.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2369-g1a6a3e74-1jammy cephadm=19.0.0-2369-g1a6a3e74-1jammy ceph-mds=19.0.0-2369-g1a6a3e74-1jammy ceph-mgr=19.0.0-2369-g1a6a3e74-1jammy ceph-common=19.0.0-2369-g1a6a3e74-1jammy ceph-fuse=19.0.0-2369-g1a6a3e74-1jammy ceph-test=19.0.0-2369-g1a6a3e74-1jammy ceph-volume=19.0.0-2369-g1a6a3e74-1jammy radosgw=19.0.0-2369-g1a6a3e74-1jammy python3-rados=19.0.0-2369-g1a6a3e74-1jammy python3-rgw=19.0.0-2369-g1a6a3e74-1jammy python3-cephfs=19.0.0-2369-g1a6a3e74-1jammy python3-rbd=19.0.0-2369-g1a6a3e74-1jammy libcephfs2=19.0.0-2369-g1a6a3e74-1jammy libcephfs-dev=19.0.0-2369-g1a6a3e74-1jammy librados2=19.0.0-2369-g1a6a3e74-1jammy librbd1=19.0.0-2369-g1a6a3e74-1jammy rbd-fuse=19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:48:32.491 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-04-18T21:48:32.553 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2024-04-18T21:48:32.575 INFO:teuthology.orchestra.run.smithi183.stdout:Building dependency tree... 2024-04-18T21:48:32.576 INFO:teuthology.orchestra.run.smithi183.stdout:Reading state information... 2024-04-18T21:48:32.689 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-04-18T21:48:32.690 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-04-18T21:48:32.751 INFO:teuthology.orchestra.run.smithi097.stdout:Building dependency tree... 2024-04-18T21:48:32.752 INFO:teuthology.orchestra.run.smithi097.stdout:Reading state information... 2024-04-18T21:48:32.770 INFO:teuthology.orchestra.run.smithi183.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:48:32.770 INFO:teuthology.orchestra.run.smithi183.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:48:32.770 INFO:teuthology.orchestra.run.smithi183.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:48:32.771 INFO:teuthology.orchestra.run.smithi183.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:48:32.771 INFO:teuthology.orchestra.run.smithi183.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T21:48:32.771 INFO:teuthology.orchestra.run.smithi183.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T21:48:32.771 INFO:teuthology.orchestra.run.smithi183.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:48:32.771 INFO:teuthology.orchestra.run.smithi183.stdout:The following additional packages will be installed: 2024-04-18T21:48:32.771 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-18T21:48:32.772 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-18T21:48:32.773 INFO:teuthology.orchestra.run.smithi183.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet zip 2024-04-18T21:48:32.774 INFO:teuthology.orchestra.run.smithi183.stdout:Suggested packages: 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout:Recommended packages: 2024-04-18T21:48:32.775 INFO:teuthology.orchestra.run.smithi183.stdout: btrfs-tools 2024-04-18T21:48:32.874 INFO:teuthology.orchestra.run.smithi183.stdout:The following NEW packages will be installed: 2024-04-18T21:48:32.874 INFO:teuthology.orchestra.run.smithi183.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-18T21:48:32.874 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-18T21:48:32.874 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-18T21:48:32.875 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-18T21:48:32.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-18T21:48:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-18T21:48:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-18T21:48:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-18T21:48:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-18T21:48:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-18T21:48:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-18T21:48:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-18T21:48:32.877 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-18T21:48:32.878 INFO:teuthology.orchestra.run.smithi183.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-18T21:48:32.887 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:48:32.887 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:48:32.887 INFO:teuthology.orchestra.run.smithi104.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:48:32.888 INFO:teuthology.orchestra.run.smithi104.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:48:32.888 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T21:48:32.888 INFO:teuthology.orchestra.run.smithi104.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T21:48:32.888 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:48:32.889 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-04-18T21:48:32.889 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-18T21:48:32.889 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-18T21:48:32.890 INFO:teuthology.orchestra.run.smithi104.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-18T21:48:32.890 INFO:teuthology.orchestra.run.smithi104.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-18T21:48:32.890 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-18T21:48:32.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet zip 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-18T21:48:32.892 INFO:teuthology.orchestra.run.smithi104.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-18T21:48:32.893 INFO:teuthology.orchestra.run.smithi104.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-18T21:48:32.893 INFO:teuthology.orchestra.run.smithi104.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-18T21:48:32.893 INFO:teuthology.orchestra.run.smithi104.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-18T21:48:32.893 INFO:teuthology.orchestra.run.smithi104.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-18T21:48:32.893 INFO:teuthology.orchestra.run.smithi104.stdout:Recommended packages: 2024-04-18T21:48:32.893 INFO:teuthology.orchestra.run.smithi104.stdout: btrfs-tools 2024-04-18T21:48:32.942 INFO:teuthology.orchestra.run.smithi183.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-18T21:48:32.942 INFO:teuthology.orchestra.run.smithi183.stdout:Need to get 179 MB of archives. 2024-04-18T21:48:32.943 INFO:teuthology.orchestra.run.smithi183.stdout:After this operation, 821 MB of additional disk space will be used. 2024-04-18T21:48:32.943 INFO:teuthology.orchestra.run.smithi183.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-18T21:48:32.949 INFO:teuthology.orchestra.run.smithi097.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T21:48:32.949 INFO:teuthology.orchestra.run.smithi097.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T21:48:32.950 INFO:teuthology.orchestra.run.smithi097.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T21:48:32.950 INFO:teuthology.orchestra.run.smithi097.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T21:48:32.950 INFO:teuthology.orchestra.run.smithi097.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T21:48:32.950 INFO:teuthology.orchestra.run.smithi097.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T21:48:32.950 INFO:teuthology.orchestra.run.smithi097.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T21:48:32.951 INFO:teuthology.orchestra.run.smithi097.stdout:The following additional packages will be installed: 2024-04-18T21:48:32.951 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-18T21:48:32.951 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-18T21:48:32.952 INFO:teuthology.orchestra.run.smithi097.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-18T21:48:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet zip 2024-04-18T21:48:32.954 INFO:teuthology.orchestra.run.smithi097.stdout:Suggested packages: 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout:Recommended packages: 2024-04-18T21:48:32.955 INFO:teuthology.orchestra.run.smithi097.stdout: btrfs-tools 2024-04-18T21:48:32.991 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-04-18T21:48:32.991 INFO:teuthology.orchestra.run.smithi104.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-18T21:48:32.991 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-18T21:48:32.991 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-18T21:48:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-18T21:48:32.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-18T21:48:32.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-18T21:48:32.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-18T21:48:32.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-18T21:48:32.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-18T21:48:32.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-18T21:48:32.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-18T21:48:32.994 INFO:teuthology.orchestra.run.smithi104.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-18T21:48:33.051 INFO:teuthology.orchestra.run.smithi183.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-18T21:48:33.054 INFO:teuthology.orchestra.run.smithi183.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-18T21:48:33.055 INFO:teuthology.orchestra.run.smithi097.stdout:The following NEW packages will be installed: 2024-04-18T21:48:33.055 INFO:teuthology.orchestra.run.smithi097.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-18T21:48:33.055 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-18T21:48:33.055 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-18T21:48:33.055 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-18T21:48:33.056 INFO:teuthology.orchestra.run.smithi097.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-18T21:48:33.056 INFO:teuthology.orchestra.run.smithi097.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-18T21:48:33.057 INFO:teuthology.orchestra.run.smithi097.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-18T21:48:33.057 INFO:teuthology.orchestra.run.smithi097.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-18T21:48:33.057 INFO:teuthology.orchestra.run.smithi097.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-18T21:48:33.057 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-18T21:48:33.057 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-18T21:48:33.057 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-18T21:48:33.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-18T21:48:33.059 INFO:teuthology.orchestra.run.smithi097.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-18T21:48:33.059 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-18T21:48:33.059 INFO:teuthology.orchestra.run.smithi097.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-18T21:48:33.059 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-18T21:48:33.059 INFO:teuthology.orchestra.run.smithi097.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-18T21:48:33.059 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-18T21:48:33.059 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 179 MB of archives. 2024-04-18T21:48:33.059 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 821 MB of additional disk space will be used. 2024-04-18T21:48:33.060 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-18T21:48:33.074 INFO:teuthology.orchestra.run.smithi183.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-18T21:48:33.126 INFO:teuthology.orchestra.run.smithi097.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-18T21:48:33.126 INFO:teuthology.orchestra.run.smithi097.stdout:Need to get 179 MB of archives. 2024-04-18T21:48:33.126 INFO:teuthology.orchestra.run.smithi097.stdout:After this operation, 821 MB of additional disk space will be used. 2024-04-18T21:48:33.126 INFO:teuthology.orchestra.run.smithi097.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-18T21:48:33.142 INFO:teuthology.orchestra.run.smithi183.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-18T21:48:33.146 INFO:teuthology.orchestra.run.smithi183.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-18T21:48:33.159 INFO:teuthology.orchestra.run.smithi183.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-18T21:48:33.163 INFO:teuthology.orchestra.run.smithi183.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-18T21:48:33.164 INFO:teuthology.orchestra.run.smithi183.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-18T21:48:33.165 INFO:teuthology.orchestra.run.smithi183.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-18T21:48:33.171 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-18T21:48:33.173 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-18T21:48:33.173 INFO:teuthology.orchestra.run.smithi183.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-18T21:48:33.176 INFO:teuthology.orchestra.run.smithi183.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-18T21:48:33.178 INFO:teuthology.orchestra.run.smithi183.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-18T21:48:33.179 INFO:teuthology.orchestra.run.smithi183.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-18T21:48:33.191 INFO:teuthology.orchestra.run.smithi183.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-18T21:48:33.193 INFO:teuthology.orchestra.run.smithi183.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-18T21:48:33.196 INFO:teuthology.orchestra.run.smithi183.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-18T21:48:33.197 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-18T21:48:33.197 INFO:teuthology.orchestra.run.smithi183.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-18T21:48:33.198 INFO:teuthology.orchestra.run.smithi183.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-18T21:48:33.199 INFO:teuthology.orchestra.run.smithi183.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,547 kB] 2024-04-18T21:48:33.212 INFO:teuthology.orchestra.run.smithi183.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-18T21:48:33.214 INFO:teuthology.orchestra.run.smithi183.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-18T21:48:33.214 INFO:teuthology.orchestra.run.smithi183.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-18T21:48:33.214 INFO:teuthology.orchestra.run.smithi183.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-18T21:48:33.215 INFO:teuthology.orchestra.run.smithi183.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-18T21:48:33.233 INFO:teuthology.orchestra.run.smithi183.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-18T21:48:33.234 INFO:teuthology.orchestra.run.smithi183.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-18T21:48:33.234 INFO:teuthology.orchestra.run.smithi183.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-18T21:48:33.236 INFO:teuthology.orchestra.run.smithi183.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-18T21:48:33.237 INFO:teuthology.orchestra.run.smithi183.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-18T21:48:33.244 INFO:teuthology.orchestra.run.smithi097.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-18T21:48:33.247 INFO:teuthology.orchestra.run.smithi097.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-18T21:48:33.254 INFO:teuthology.orchestra.run.smithi183.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-18T21:48:33.254 INFO:teuthology.orchestra.run.smithi183.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-18T21:48:33.255 INFO:teuthology.orchestra.run.smithi183.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-18T21:48:33.256 INFO:teuthology.orchestra.run.smithi183.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-18T21:48:33.256 INFO:teuthology.orchestra.run.smithi183.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-18T21:48:33.266 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-18T21:48:33.271 INFO:teuthology.orchestra.run.smithi097.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-18T21:48:33.271 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-18T21:48:33.279 INFO:teuthology.orchestra.run.smithi183.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-18T21:48:33.280 INFO:teuthology.orchestra.run.smithi183.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-18T21:48:33.280 INFO:teuthology.orchestra.run.smithi183.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-18T21:48:33.284 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-18T21:48:33.286 INFO:teuthology.orchestra.run.smithi183.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-18T21:48:33.286 INFO:teuthology.orchestra.run.smithi183.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-18T21:48:33.288 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-18T21:48:33.289 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-18T21:48:33.290 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-18T21:48:33.298 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-18T21:48:33.301 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-18T21:48:33.303 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-18T21:48:33.303 INFO:teuthology.orchestra.run.smithi183.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-18T21:48:33.304 INFO:teuthology.orchestra.run.smithi183.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-18T21:48:33.305 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-18T21:48:33.305 INFO:teuthology.orchestra.run.smithi183.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-18T21:48:33.307 INFO:teuthology.orchestra.run.smithi183.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-18T21:48:33.308 INFO:teuthology.orchestra.run.smithi183.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-18T21:48:33.321 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-18T21:48:33.323 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-18T21:48:33.325 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-18T21:48:33.327 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-18T21:48:33.328 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-18T21:48:33.328 INFO:teuthology.orchestra.run.smithi183.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-18T21:48:33.329 INFO:teuthology.orchestra.run.smithi183.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-18T21:48:33.331 INFO:teuthology.orchestra.run.smithi183.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-18T21:48:33.332 INFO:teuthology.orchestra.run.smithi183.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-18T21:48:33.333 INFO:teuthology.orchestra.run.smithi183.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-18T21:48:33.346 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-18T21:48:33.347 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-18T21:48:33.348 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-18T21:48:33.348 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-18T21:48:33.348 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-18T21:48:33.354 INFO:teuthology.orchestra.run.smithi183.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-18T21:48:33.359 INFO:teuthology.orchestra.run.smithi097.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-18T21:48:33.363 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-18T21:48:33.389 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-18T21:48:33.396 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-18T21:48:33.397 INFO:teuthology.orchestra.run.smithi097.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-18T21:48:33.397 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-18T21:48:33.397 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-18T21:48:33.398 INFO:teuthology.orchestra.run.smithi097.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-18T21:48:33.398 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-18T21:48:33.403 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-18T21:48:33.403 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-18T21:48:33.410 INFO:teuthology.orchestra.run.smithi097.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-18T21:48:33.413 INFO:teuthology.orchestra.run.smithi097.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-18T21:48:33.416 INFO:teuthology.orchestra.run.smithi097.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-18T21:48:33.418 INFO:teuthology.orchestra.run.smithi097.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-18T21:48:33.435 INFO:teuthology.orchestra.run.smithi183.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-18T21:48:33.437 INFO:teuthology.orchestra.run.smithi183.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-18T21:48:33.438 INFO:teuthology.orchestra.run.smithi183.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-18T21:48:33.439 INFO:teuthology.orchestra.run.smithi183.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-18T21:48:33.440 INFO:teuthology.orchestra.run.smithi183.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-18T21:48:33.443 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,547 kB] 2024-04-18T21:48:33.447 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-18T21:48:33.447 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-18T21:48:33.449 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-18T21:48:33.450 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-18T21:48:33.451 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-18T21:48:33.461 INFO:teuthology.orchestra.run.smithi097.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-18T21:48:33.465 INFO:teuthology.orchestra.run.smithi097.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-18T21:48:33.468 INFO:teuthology.orchestra.run.smithi097.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-18T21:48:33.472 INFO:teuthology.orchestra.run.smithi097.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-18T21:48:33.473 INFO:teuthology.orchestra.run.smithi097.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-18T21:48:33.473 INFO:teuthology.orchestra.run.smithi097.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-18T21:48:33.476 INFO:teuthology.orchestra.run.smithi183.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-18T21:48:33.479 INFO:teuthology.orchestra.run.smithi183.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-18T21:48:33.479 INFO:teuthology.orchestra.run.smithi183.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-18T21:48:33.494 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-18T21:48:33.495 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-18T21:48:33.496 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-18T21:48:33.505 INFO:teuthology.orchestra.run.smithi183.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-18T21:48:33.505 INFO:teuthology.orchestra.run.smithi183.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-18T21:48:33.506 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-18T21:48:33.507 INFO:teuthology.orchestra.run.smithi097.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-18T21:48:33.507 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-18T21:48:33.508 INFO:teuthology.orchestra.run.smithi097.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-18T21:48:33.508 INFO:teuthology.orchestra.run.smithi097.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-18T21:48:33.509 INFO:teuthology.orchestra.run.smithi097.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-18T21:48:33.512 INFO:teuthology.orchestra.run.smithi183.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-18T21:48:33.512 INFO:teuthology.orchestra.run.smithi183.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-18T21:48:33.513 INFO:teuthology.orchestra.run.smithi183.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-18T21:48:33.513 INFO:teuthology.orchestra.run.smithi183.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-18T21:48:33.514 INFO:teuthology.orchestra.run.smithi183.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-18T21:48:33.514 INFO:teuthology.orchestra.run.smithi183.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-18T21:48:33.518 INFO:teuthology.orchestra.run.smithi183.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-18T21:48:33.521 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-18T21:48:33.522 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-18T21:48:33.524 INFO:teuthology.orchestra.run.smithi097.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,547 kB] 2024-04-18T21:48:33.524 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-18T21:48:33.528 INFO:teuthology.orchestra.run.smithi104.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-18T21:48:33.530 INFO:teuthology.orchestra.run.smithi104.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-18T21:48:33.530 INFO:teuthology.orchestra.run.smithi097.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-18T21:48:33.530 INFO:teuthology.orchestra.run.smithi097.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-18T21:48:33.531 INFO:teuthology.orchestra.run.smithi097.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-18T21:48:33.534 INFO:teuthology.orchestra.run.smithi183.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-18T21:48:33.537 INFO:teuthology.orchestra.run.smithi097.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-18T21:48:33.537 INFO:teuthology.orchestra.run.smithi183.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-18T21:48:33.538 INFO:teuthology.orchestra.run.smithi097.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-18T21:48:33.538 INFO:teuthology.orchestra.run.smithi097.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-18T21:48:33.540 INFO:teuthology.orchestra.run.smithi183.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-18T21:48:33.543 INFO:teuthology.orchestra.run.smithi104.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-18T21:48:33.544 INFO:teuthology.orchestra.run.smithi104.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-18T21:48:33.549 INFO:teuthology.orchestra.run.smithi104.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-18T21:48:33.550 INFO:teuthology.orchestra.run.smithi104.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-18T21:48:33.551 INFO:teuthology.orchestra.run.smithi097.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-18T21:48:33.552 INFO:teuthology.orchestra.run.smithi104.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-18T21:48:33.553 INFO:teuthology.orchestra.run.smithi097.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-18T21:48:33.553 INFO:teuthology.orchestra.run.smithi097.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-18T21:48:33.554 INFO:teuthology.orchestra.run.smithi097.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-18T21:48:33.555 INFO:teuthology.orchestra.run.smithi183.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-18T21:48:33.559 INFO:teuthology.orchestra.run.smithi183.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-18T21:48:33.563 INFO:teuthology.orchestra.run.smithi183.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-18T21:48:33.566 INFO:teuthology.orchestra.run.smithi183.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-18T21:48:33.567 INFO:teuthology.orchestra.run.smithi104.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-18T21:48:33.571 INFO:teuthology.orchestra.run.smithi183.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-18T21:48:33.572 INFO:teuthology.orchestra.run.smithi183.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-18T21:48:33.575 INFO:teuthology.orchestra.run.smithi183.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-18T21:48:33.582 INFO:teuthology.orchestra.run.smithi097.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-18T21:48:33.584 INFO:teuthology.orchestra.run.smithi097.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-18T21:48:33.585 INFO:teuthology.orchestra.run.smithi097.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-18T21:48:33.595 INFO:teuthology.orchestra.run.smithi183.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-18T21:48:33.596 INFO:teuthology.orchestra.run.smithi183.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-18T21:48:33.597 INFO:teuthology.orchestra.run.smithi097.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-18T21:48:33.597 INFO:teuthology.orchestra.run.smithi183.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-18T21:48:33.597 INFO:teuthology.orchestra.run.smithi097.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-18T21:48:33.598 INFO:teuthology.orchestra.run.smithi097.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-18T21:48:33.619 INFO:teuthology.orchestra.run.smithi097.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-18T21:48:33.622 INFO:teuthology.orchestra.run.smithi097.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-18T21:48:33.626 INFO:teuthology.orchestra.run.smithi097.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-18T21:48:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-18T21:48:33.633 INFO:teuthology.orchestra.run.smithi183.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-18T21:48:33.634 INFO:teuthology.orchestra.run.smithi183.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-18T21:48:33.643 INFO:teuthology.orchestra.run.smithi183.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-18T21:48:33.644 INFO:teuthology.orchestra.run.smithi183.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-18T21:48:33.644 INFO:teuthology.orchestra.run.smithi183.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-18T21:48:33.647 INFO:teuthology.orchestra.run.smithi183.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-18T21:48:33.647 INFO:teuthology.orchestra.run.smithi183.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-18T21:48:33.651 INFO:teuthology.orchestra.run.smithi104.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-18T21:48:33.653 INFO:teuthology.orchestra.run.smithi104.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-18T21:48:33.660 INFO:teuthology.orchestra.run.smithi104.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-18T21:48:33.660 INFO:teuthology.orchestra.run.smithi104.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-18T21:48:33.662 INFO:teuthology.orchestra.run.smithi104.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-18T21:48:33.681 INFO:teuthology.orchestra.run.smithi097.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-18T21:48:33.682 INFO:teuthology.orchestra.run.smithi097.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-18T21:48:33.688 INFO:teuthology.orchestra.run.smithi097.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-18T21:48:33.689 INFO:teuthology.orchestra.run.smithi097.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-18T21:48:33.691 INFO:teuthology.orchestra.run.smithi097.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-18T21:48:33.697 INFO:teuthology.orchestra.run.smithi097.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-18T21:48:33.769 INFO:teuthology.orchestra.run.smithi104.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-18T21:48:33.774 INFO:teuthology.orchestra.run.smithi104.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-18T21:48:33.775 INFO:teuthology.orchestra.run.smithi104.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-18T21:48:33.809 INFO:teuthology.orchestra.run.smithi104.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-18T21:48:33.809 INFO:teuthology.orchestra.run.smithi104.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-18T21:48:33.810 INFO:teuthology.orchestra.run.smithi104.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-18T21:48:33.811 INFO:teuthology.orchestra.run.smithi104.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-18T21:48:33.812 INFO:teuthology.orchestra.run.smithi104.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-18T21:48:33.812 INFO:teuthology.orchestra.run.smithi104.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-18T21:48:33.813 INFO:teuthology.orchestra.run.smithi104.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-18T21:48:33.818 INFO:teuthology.orchestra.run.smithi104.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-18T21:48:33.823 INFO:teuthology.orchestra.run.smithi104.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-18T21:48:33.826 INFO:teuthology.orchestra.run.smithi097.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-18T21:48:33.827 INFO:teuthology.orchestra.run.smithi097.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-18T21:48:33.828 INFO:teuthology.orchestra.run.smithi097.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-18T21:48:33.828 INFO:teuthology.orchestra.run.smithi097.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-18T21:48:33.830 INFO:teuthology.orchestra.run.smithi097.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-18T21:48:33.839 INFO:teuthology.orchestra.run.smithi104.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-18T21:48:33.841 INFO:teuthology.orchestra.run.smithi104.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-18T21:48:33.842 INFO:teuthology.orchestra.run.smithi104.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-18T21:48:33.847 INFO:teuthology.orchestra.run.smithi104.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-18T21:48:33.850 INFO:teuthology.orchestra.run.smithi104.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-18T21:48:33.852 INFO:teuthology.orchestra.run.smithi104.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-18T21:48:33.852 INFO:teuthology.orchestra.run.smithi104.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-18T21:48:33.859 INFO:teuthology.orchestra.run.smithi104.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-18T21:48:33.867 INFO:teuthology.orchestra.run.smithi104.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-18T21:48:33.870 INFO:teuthology.orchestra.run.smithi097.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-18T21:48:33.872 INFO:teuthology.orchestra.run.smithi104.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-18T21:48:33.874 INFO:teuthology.orchestra.run.smithi097.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-18T21:48:33.874 INFO:teuthology.orchestra.run.smithi097.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-18T21:48:33.891 INFO:teuthology.orchestra.run.smithi183.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,168 kB] 2024-04-18T21:48:33.894 INFO:teuthology.orchestra.run.smithi104.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-18T21:48:33.895 INFO:teuthology.orchestra.run.smithi104.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-18T21:48:33.896 INFO:teuthology.orchestra.run.smithi104.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-18T21:48:33.898 INFO:teuthology.orchestra.run.smithi104.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-18T21:48:33.900 INFO:teuthology.orchestra.run.smithi104.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-18T21:48:33.902 INFO:teuthology.orchestra.run.smithi097.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-18T21:48:33.903 INFO:teuthology.orchestra.run.smithi097.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-18T21:48:33.904 INFO:teuthology.orchestra.run.smithi097.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-18T21:48:33.906 INFO:teuthology.orchestra.run.smithi097.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-18T21:48:33.906 INFO:teuthology.orchestra.run.smithi097.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-18T21:48:33.908 INFO:teuthology.orchestra.run.smithi097.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-18T21:48:33.910 INFO:teuthology.orchestra.run.smithi104.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-18T21:48:33.910 INFO:teuthology.orchestra.run.smithi104.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-18T21:48:33.911 INFO:teuthology.orchestra.run.smithi104.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-18T21:48:33.914 INFO:teuthology.orchestra.run.smithi097.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-18T21:48:33.915 INFO:teuthology.orchestra.run.smithi097.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-18T21:48:33.919 INFO:teuthology.orchestra.run.smithi104.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-18T21:48:33.920 INFO:teuthology.orchestra.run.smithi097.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-18T21:48:33.920 INFO:teuthology.orchestra.run.smithi104.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-18T21:48:33.941 INFO:teuthology.orchestra.run.smithi097.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-18T21:48:33.943 INFO:teuthology.orchestra.run.smithi097.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-18T21:48:33.944 INFO:teuthology.orchestra.run.smithi097.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-18T21:48:33.948 INFO:teuthology.orchestra.run.smithi097.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-18T21:48:33.952 INFO:teuthology.orchestra.run.smithi097.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-18T21:48:33.953 INFO:teuthology.orchestra.run.smithi097.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-18T21:48:33.954 INFO:teuthology.orchestra.run.smithi097.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-18T21:48:33.962 INFO:teuthology.orchestra.run.smithi097.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-18T21:48:33.963 INFO:teuthology.orchestra.run.smithi097.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-18T21:48:33.967 INFO:teuthology.orchestra.run.smithi097.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-18T21:48:33.983 INFO:teuthology.orchestra.run.smithi097.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-18T21:48:33.984 INFO:teuthology.orchestra.run.smithi097.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-18T21:48:33.984 INFO:teuthology.orchestra.run.smithi097.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-18T21:48:33.987 INFO:teuthology.orchestra.run.smithi097.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-18T21:48:33.988 INFO:teuthology.orchestra.run.smithi097.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-18T21:48:33.998 INFO:teuthology.orchestra.run.smithi097.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-18T21:48:33.998 INFO:teuthology.orchestra.run.smithi097.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-18T21:48:34.004 INFO:teuthology.orchestra.run.smithi097.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-18T21:48:34.007 INFO:teuthology.orchestra.run.smithi097.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-18T21:48:34.008 INFO:teuthology.orchestra.run.smithi097.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-18T21:48:34.250 INFO:teuthology.orchestra.run.smithi183.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2369-g1a6a3e74-1jammy [711 kB] 2024-04-18T21:48:34.425 INFO:teuthology.orchestra.run.smithi183.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2369-g1a6a3e74-1jammy [356 kB] 2024-04-18T21:48:34.436 INFO:teuthology.orchestra.run.smithi183.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2369-g1a6a3e74-1jammy [32.8 kB] 2024-04-18T21:48:34.438 INFO:teuthology.orchestra.run.smithi183.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2369-g1a6a3e74-1jammy [180 kB] 2024-04-18T21:48:34.442 INFO:teuthology.orchestra.run.smithi183.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2369-g1a6a3e74-1jammy [65.7 kB] 2024-04-18T21:48:34.444 INFO:teuthology.orchestra.run.smithi183.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2369-g1a6a3e74-1jammy [330 kB] 2024-04-18T21:48:34.466 INFO:teuthology.orchestra.run.smithi183.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2369-g1a6a3e74-1jammy [6,775 kB] 2024-04-18T21:48:34.865 INFO:teuthology.orchestra.run.smithi104.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,168 kB] 2024-04-18T21:48:35.193 INFO:teuthology.orchestra.run.smithi183.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2369-g1a6a3e74-1jammy [112 kB] 2024-04-18T21:48:35.202 INFO:teuthology.orchestra.run.smithi183.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2369-g1a6a3e74-1jammy [466 kB] 2024-04-18T21:48:35.225 INFO:teuthology.orchestra.run.smithi183.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2369-g1a6a3e74-1jammy [25.6 MB] 2024-04-18T21:48:35.322 INFO:teuthology.orchestra.run.smithi104.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2369-g1a6a3e74-1jammy [711 kB] 2024-04-18T21:48:35.399 INFO:teuthology.orchestra.run.smithi104.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2369-g1a6a3e74-1jammy [356 kB] 2024-04-18T21:48:35.424 INFO:teuthology.orchestra.run.smithi104.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2369-g1a6a3e74-1jammy [32.8 kB] 2024-04-18T21:48:35.426 INFO:teuthology.orchestra.run.smithi104.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2369-g1a6a3e74-1jammy [180 kB] 2024-04-18T21:48:35.444 INFO:teuthology.orchestra.run.smithi104.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2369-g1a6a3e74-1jammy [65.7 kB] 2024-04-18T21:48:35.449 INFO:teuthology.orchestra.run.smithi104.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2369-g1a6a3e74-1jammy [330 kB] 2024-04-18T21:48:35.468 INFO:teuthology.orchestra.run.smithi097.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,168 kB] 2024-04-18T21:48:35.486 INFO:teuthology.orchestra.run.smithi104.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2369-g1a6a3e74-1jammy [6,775 kB] 2024-04-18T21:48:36.039 INFO:teuthology.orchestra.run.smithi104.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2369-g1a6a3e74-1jammy [112 kB] 2024-04-18T21:48:36.043 INFO:teuthology.orchestra.run.smithi104.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2369-g1a6a3e74-1jammy [466 kB] 2024-04-18T21:48:36.058 INFO:teuthology.orchestra.run.smithi097.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2369-g1a6a3e74-1jammy [711 kB] 2024-04-18T21:48:36.088 INFO:teuthology.orchestra.run.smithi104.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2369-g1a6a3e74-1jammy [25.6 MB] 2024-04-18T21:48:36.141 INFO:teuthology.orchestra.run.smithi097.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2369-g1a6a3e74-1jammy [356 kB] 2024-04-18T21:48:36.208 INFO:teuthology.orchestra.run.smithi097.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2369-g1a6a3e74-1jammy [32.8 kB] 2024-04-18T21:48:36.209 INFO:teuthology.orchestra.run.smithi097.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2369-g1a6a3e74-1jammy [180 kB] 2024-04-18T21:48:36.270 INFO:teuthology.orchestra.run.smithi097.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2369-g1a6a3e74-1jammy [65.7 kB] 2024-04-18T21:48:36.272 INFO:teuthology.orchestra.run.smithi097.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2369-g1a6a3e74-1jammy [330 kB] 2024-04-18T21:48:36.289 INFO:teuthology.orchestra.run.smithi097.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2369-g1a6a3e74-1jammy [6,775 kB] 2024-04-18T21:48:37.331 INFO:teuthology.orchestra.run.smithi097.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2369-g1a6a3e74-1jammy [112 kB] 2024-04-18T21:48:37.334 INFO:teuthology.orchestra.run.smithi097.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2369-g1a6a3e74-1jammy [466 kB] 2024-04-18T21:48:37.405 INFO:teuthology.orchestra.run.smithi097.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2369-g1a6a3e74-1jammy [25.6 MB] 2024-04-18T21:48:37.858 INFO:teuthology.orchestra.run.smithi104.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2369-g1a6a3e74-1jammy [5,084 kB] 2024-04-18T21:48:37.887 INFO:teuthology.orchestra.run.smithi183.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2369-g1a6a3e74-1jammy [5,084 kB] 2024-04-18T21:48:38.084 INFO:teuthology.orchestra.run.smithi104.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2369-g1a6a3e74-1jammy [244 kB] 2024-04-18T21:48:38.098 INFO:teuthology.orchestra.run.smithi104.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [124 kB] 2024-04-18T21:48:38.102 INFO:teuthology.orchestra.run.smithi104.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2369-g1a6a3e74-1jammy [1,525 kB] 2024-04-18T21:48:38.215 INFO:teuthology.orchestra.run.smithi104.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2369-g1a6a3e74-1jammy [6,246 kB] 2024-04-18T21:48:38.413 INFO:teuthology.orchestra.run.smithi183.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2369-g1a6a3e74-1jammy [244 kB] 2024-04-18T21:48:38.448 INFO:teuthology.orchestra.run.smithi183.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [124 kB] 2024-04-18T21:48:38.464 INFO:teuthology.orchestra.run.smithi183.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2369-g1a6a3e74-1jammy [1,525 kB] 2024-04-18T21:48:38.626 INFO:teuthology.orchestra.run.smithi104.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2369-g1a6a3e74-1jammy [22.7 MB] 2024-04-18T21:48:38.665 INFO:teuthology.orchestra.run.smithi183.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2369-g1a6a3e74-1jammy [6,246 kB] 2024-04-18T21:48:39.516 INFO:teuthology.orchestra.run.smithi183.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2369-g1a6a3e74-1jammy [22.7 MB] 2024-04-18T21:48:40.101 INFO:teuthology.orchestra.run.smithi104.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [14.0 kB] 2024-04-18T21:48:40.102 INFO:teuthology.orchestra.run.smithi104.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [869 kB] 2024-04-18T21:48:40.119 INFO:teuthology.orchestra.run.smithi104.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2369-g1a6a3e74-1jammy [2,435 kB] 2024-04-18T21:48:40.251 INFO:teuthology.orchestra.run.smithi104.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2369-g1a6a3e74-1jammy [791 kB] 2024-04-18T21:48:40.288 INFO:teuthology.orchestra.run.smithi104.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2369-g1a6a3e74-1jammy [147 kB] 2024-04-18T21:48:40.293 INFO:teuthology.orchestra.run.smithi104.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2369-g1a6a3e74-1jammy [2,295 kB] 2024-04-18T21:48:40.445 INFO:teuthology.orchestra.run.smithi104.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2369-g1a6a3e74-1jammy [8,625 kB] 2024-04-18T21:48:40.964 INFO:teuthology.orchestra.run.smithi097.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2369-g1a6a3e74-1jammy [5,084 kB] 2024-04-18T21:48:40.984 INFO:teuthology.orchestra.run.smithi104.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2369-g1a6a3e74-1jammy [14.2 kB] 2024-04-18T21:48:40.985 INFO:teuthology.orchestra.run.smithi104.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2369-g1a6a3e74-1jammy [50.7 MB] 2024-04-18T21:48:41.558 INFO:teuthology.orchestra.run.smithi097.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2369-g1a6a3e74-1jammy [244 kB] 2024-04-18T21:48:41.577 INFO:teuthology.orchestra.run.smithi097.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [124 kB] 2024-04-18T21:48:41.610 INFO:teuthology.orchestra.run.smithi097.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2369-g1a6a3e74-1jammy [1,525 kB] 2024-04-18T21:48:41.813 INFO:teuthology.orchestra.run.smithi097.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2369-g1a6a3e74-1jammy [6,246 kB] 2024-04-18T21:48:42.617 INFO:teuthology.orchestra.run.smithi097.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2369-g1a6a3e74-1jammy [22.7 MB] 2024-04-18T21:48:43.325 INFO:teuthology.orchestra.run.smithi183.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [14.0 kB] 2024-04-18T21:48:43.326 INFO:teuthology.orchestra.run.smithi183.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [869 kB] 2024-04-18T21:48:43.355 INFO:teuthology.orchestra.run.smithi183.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2369-g1a6a3e74-1jammy [2,435 kB] 2024-04-18T21:48:43.786 INFO:teuthology.orchestra.run.smithi183.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2369-g1a6a3e74-1jammy [791 kB] 2024-04-18T21:48:43.923 INFO:teuthology.orchestra.run.smithi183.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2369-g1a6a3e74-1jammy [147 kB] 2024-04-18T21:48:43.953 INFO:teuthology.orchestra.run.smithi183.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2369-g1a6a3e74-1jammy [2,295 kB] 2024-04-18T21:48:44.216 INFO:teuthology.orchestra.run.smithi104.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2369-g1a6a3e74-1jammy [121 kB] 2024-04-18T21:48:44.218 INFO:teuthology.orchestra.run.smithi104.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2369-g1a6a3e74-1jammy [38.9 kB] 2024-04-18T21:48:44.219 INFO:teuthology.orchestra.run.smithi104.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2369-g1a6a3e74-1jammy [13.4 MB] 2024-04-18T21:48:44.319 INFO:teuthology.orchestra.run.smithi183.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2369-g1a6a3e74-1jammy [8,625 kB] 2024-04-18T21:48:44.933 INFO:teuthology.orchestra.run.smithi104.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [91.4 kB] 2024-04-18T21:48:45.136 INFO:teuthology.orchestra.run.smithi097.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [14.0 kB] 2024-04-18T21:48:45.137 INFO:teuthology.orchestra.run.smithi097.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [869 kB] 2024-04-18T21:48:45.154 INFO:teuthology.orchestra.run.smithi097.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2369-g1a6a3e74-1jammy [2,435 kB] 2024-04-18T21:48:45.257 INFO:teuthology.orchestra.run.smithi183.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2369-g1a6a3e74-1jammy [14.2 kB] 2024-04-18T21:48:45.258 INFO:teuthology.orchestra.run.smithi183.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2369-g1a6a3e74-1jammy [50.7 MB] 2024-04-18T21:48:45.269 INFO:teuthology.orchestra.run.smithi097.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2369-g1a6a3e74-1jammy [791 kB] 2024-04-18T21:48:45.314 INFO:teuthology.orchestra.run.smithi097.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2369-g1a6a3e74-1jammy [147 kB] 2024-04-18T21:48:45.323 INFO:teuthology.orchestra.run.smithi097.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2369-g1a6a3e74-1jammy [2,295 kB] 2024-04-18T21:48:45.458 INFO:teuthology.orchestra.run.smithi097.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2369-g1a6a3e74-1jammy [8,625 kB] 2024-04-18T21:48:45.503 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 179 MB in 12s (15.0 MB/s) 2024-04-18T21:48:45.668 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-18T21:48:45.758 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-18T21:48:45.764 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-18T21:48:45.850 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T21:48:45.976 INFO:teuthology.orchestra.run.smithi097.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2369-g1a6a3e74-1jammy [14.2 kB] 2024-04-18T21:48:45.976 INFO:teuthology.orchestra.run.smithi097.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2369-g1a6a3e74-1jammy [50.7 MB] 2024-04-18T21:48:46.187 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-18T21:48:46.214 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-18T21:48:46.254 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T21:48:46.540 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-18T21:48:46.567 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-18T21:48:46.598 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T21:48:47.027 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-18T21:48:47.054 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:47.101 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:47.488 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-18T21:48:47.514 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:47.554 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:47.866 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-18T21:48:47.893 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:47.924 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:48.294 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-18T21:48:48.320 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-18T21:48:48.360 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T21:48:48.880 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librados2. 2024-04-18T21:48:48.907 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../007-librados2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:48.947 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:49.122 INFO:teuthology.orchestra.run.smithi183.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2369-g1a6a3e74-1jammy [121 kB] 2024-04-18T21:48:49.124 INFO:teuthology.orchestra.run.smithi183.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2369-g1a6a3e74-1jammy [38.9 kB] 2024-04-18T21:48:49.125 INFO:teuthology.orchestra.run.smithi183.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2369-g1a6a3e74-1jammy [13.4 MB] 2024-04-18T21:48:49.293 INFO:teuthology.orchestra.run.smithi097.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2369-g1a6a3e74-1jammy [121 kB] 2024-04-18T21:48:49.295 INFO:teuthology.orchestra.run.smithi097.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2369-g1a6a3e74-1jammy [38.9 kB] 2024-04-18T21:48:49.296 INFO:teuthology.orchestra.run.smithi097.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2369-g1a6a3e74-1jammy [13.4 MB] 2024-04-18T21:48:49.366 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librbd1. 2024-04-18T21:48:49.393 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../008-librbd1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:49.424 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:49.860 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libcephfs2. 2024-04-18T21:48:49.887 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:49.927 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:49.942 INFO:teuthology.orchestra.run.smithi097.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [91.4 kB] 2024-04-18T21:48:49.963 INFO:teuthology.orchestra.run.smithi183.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [91.4 kB] 2024-04-18T21:48:50.204 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rados. 2024-04-18T21:48:50.220 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:50.262 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:50.494 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 179 MB in 17s (10.6 MB/s) 2024-04-18T21:48:50.511 INFO:teuthology.orchestra.run.smithi183.stdout:Fetched 179 MB in 17s (10.5 MB/s) 2024-04-18T21:48:50.590 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-18T21:48:50.617 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:48:50.647 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-18T21:48:50.657 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-18T21:48:50.743 INFO:teuthology.orchestra.run.smithi183.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-18T21:48:50.748 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-18T21:48:50.761 INFO:teuthology.orchestra.run.smithi097.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-18T21:48:50.766 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-18T21:48:50.820 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T21:48:50.850 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T21:48:50.909 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cephfs. 2024-04-18T21:48:50.927 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:50.959 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:51.123 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-18T21:48:51.150 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-18T21:48:51.190 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T21:48:51.195 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-18T21:48:51.221 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-18T21:48:51.262 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T21:48:51.270 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-18T21:48:51.296 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:48:51.337 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:51.534 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-18T21:48:51.561 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-18T21:48:51.601 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T21:48:51.606 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-18T21:48:51.633 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-18T21:48:51.673 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T21:48:51.673 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-18T21:48:51.699 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-18T21:48:51.740 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T21:48:52.004 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-18T21:48:52.031 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:52.059 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-prettytable. 2024-04-18T21:48:52.071 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:52.085 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-18T21:48:52.126 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-18T21:48:52.128 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-18T21:48:52.155 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:52.203 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:52.353 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rbd. 2024-04-18T21:48:52.369 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:52.403 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:52.449 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-18T21:48:52.476 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:52.549 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:52.615 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-18T21:48:52.641 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:52.681 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:52.723 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-18T21:48:52.749 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-18T21:48:52.798 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T21:48:52.902 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-18T21:48:52.928 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:52.968 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:52.993 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-18T21:48:53.019 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T21:48:53.051 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:48:53.075 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-18T21:48:53.102 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-18T21:48:53.142 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T21:48:53.288 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-18T21:48:53.315 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-18T21:48:53.347 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T21:48:53.387 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-18T21:48:53.413 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-18T21:48:53.444 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-18T21:48:53.454 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T21:48:53.471 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-18T21:48:53.511 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T21:48:53.616 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package librados2. 2024-04-18T21:48:53.642 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../007-librados2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:53.682 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:53.782 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package librados2. 2024-04-18T21:48:53.808 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../007-librados2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua5.1. 2024-04-18T21:48:53.848 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:53.857 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-18T21:48:53.898 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-18T21:48:54.152 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package librbd1. 2024-04-18T21:48:54.179 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../008-librbd1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:54.227 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:54.326 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package librbd1. 2024-04-18T21:48:54.334 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-any. 2024-04-18T21:48:54.349 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-18T21:48:54.352 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../008-librbd1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:54.392 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-18T21:48:54.401 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:54.636 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package zip. 2024-04-18T21:48:54.652 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-18T21:48:54.680 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libcephfs2. 2024-04-18T21:48:54.695 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking zip (3.0-12build2) ... 2024-04-18T21:48:54.706 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:54.746 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:54.787 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libcephfs2. 2024-04-18T21:48:54.813 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:54.845 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:54.990 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-rados. 2024-04-18T21:48:54.998 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package unzip. 2024-04-18T21:48:55.003 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:55.024 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-18T21:48:55.032 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:55.064 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-18T21:48:55.114 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-rados. 2024-04-18T21:48:55.129 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:55.172 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:55.343 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-18T21:48:55.369 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:48:55.409 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:55.426 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package luarocks. 2024-04-18T21:48:55.453 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-18T21:48:55.493 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-18T21:48:55.670 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cephfs. 2024-04-18T21:48:55.685 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:55.700 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-18T21:48:55.726 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:48:55.728 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:55.766 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:55.876 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librgw2. 2024-04-18T21:48:55.902 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../025-librgw2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:55.942 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:56.019 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cephfs. 2024-04-18T21:48:56.035 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:56.048 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-18T21:48:56.074 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:48:56.077 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:56.114 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:56.338 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-18T21:48:56.353 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:48:56.388 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:56.426 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-18T21:48:56.452 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-18T21:48:56.492 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T21:48:56.511 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rgw. 2024-04-18T21:48:56.538 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:56.578 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-18T21:48:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-18T21:48:56.762 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-prettytable. 2024-04-18T21:48:56.774 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T21:48:56.788 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-18T21:48:56.820 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-18T21:48:56.872 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-18T21:48:56.899 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-18T21:48:56.930 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T21:48:57.053 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-prettytable. 2024-04-18T21:48:57.064 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-rbd. 2024-04-18T21:48:57.078 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-18T21:48:57.079 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:57.119 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-18T21:48:57.122 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:57.234 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libradosstriper1. 2024-04-18T21:48:57.260 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:57.300 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:57.388 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-rbd. 2024-04-18T21:48:57.403 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:57.446 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:57.475 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-18T21:48:57.502 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-18T21:48:57.542 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T21:48:57.728 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-common. 2024-04-18T21:48:57.755 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:57.794 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:57.816 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-18T21:48:57.819 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-18T21:48:57.842 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-18T21:48:57.846 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-18T21:48:57.882 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T21:48:57.886 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T21:48:58.152 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-18T21:48:58.167 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-18T21:48:58.172 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-18T21:48:58.198 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-18T21:48:58.201 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T21:48:58.239 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T21:48:58.479 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-18T21:48:58.494 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-18T21:48:58.529 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T21:48:58.575 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package lua5.1. 2024-04-18T21:48:58.602 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-18T21:48:58.642 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-18T21:48:58.857 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package lua5.1. 2024-04-18T21:48:58.884 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-18T21:48:58.924 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-18T21:48:58.978 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package lua-any. 2024-04-18T21:48:58.994 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-18T21:48:59.028 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-18T21:48:59.175 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-base. 2024-04-18T21:48:59.201 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:48:59.250 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:48:59.260 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package lua-any. 2024-04-18T21:48:59.275 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-18T21:48:59.318 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-18T21:48:59.364 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package zip. 2024-04-18T21:48:59.380 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-18T21:48:59.431 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking zip (3.0-12build2) ... 2024-04-18T21:48:59.538 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package zip. 2024-04-18T21:48:59.553 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-18T21:48:59.587 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking zip (3.0-12build2) ... 2024-04-18T21:48:59.742 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package unzip. 2024-04-18T21:48:59.769 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-18T21:48:59.809 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-18T21:48:59.814 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-18T21:48:59.828 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-18T21:48:59.862 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-18T21:48:59.882 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package unzip. 2024-04-18T21:48:59.908 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-18T21:48:59.949 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-18T21:49:00.128 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cheroot. 2024-04-18T21:49:00.145 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-18T21:49:00.186 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T21:49:00.213 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package luarocks. 2024-04-18T21:49:00.240 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-18T21:49:00.279 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-18T21:49:00.345 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package luarocks. 2024-04-18T21:49:00.370 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-18T21:49:00.411 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-18T21:49:00.489 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-18T21:49:00.515 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-18T21:49:00.555 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-18T21:49:00.636 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package librgw2. 2024-04-18T21:49:00.663 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../025-librgw2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:00.712 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:00.825 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-18T21:49:00.851 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-18T21:49:00.891 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-18T21:49:00.951 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package librgw2. 2024-04-18T21:49:00.978 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../025-librgw2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:01.018 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:01.194 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-18T21:49:01.221 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-18T21:49:01.239 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-rgw. 2024-04-18T21:49:01.254 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:01.261 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-18T21:49:01.289 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:01.554 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-rgw. 2024-04-18T21:49:01.564 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-tempora. 2024-04-18T21:49:01.569 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:01.590 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-18T21:49:01.604 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:01.625 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-18T21:49:01.630 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-18T21:49:01.652 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-18T21:49:01.692 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T21:49:01.900 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-portend. 2024-04-18T21:49:01.923 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-18T21:49:01.926 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-18T21:49:01.949 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-18T21:49:01.958 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-18T21:49:01.990 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T21:49:02.011 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libradosstriper1. 2024-04-18T21:49:02.037 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:02.078 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:02.278 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-18T21:49:02.301 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libradosstriper1. 2024-04-18T21:49:02.304 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-18T21:49:02.327 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:02.344 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-18T21:49:02.367 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:02.440 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-common. 2024-04-18T21:49:02.466 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:02.497 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:02.647 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-18T21:49:02.673 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-18T21:49:02.714 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-18T21:49:02.796 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-common. 2024-04-18T21:49:02.822 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:02.862 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:03.059 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-natsort. 2024-04-18T21:49:03.085 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-18T21:49:03.125 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-18T21:49:03.445 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-logutils. 2024-04-18T21:49:03.471 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-18T21:49:03.511 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-18T21:49:03.773 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-18T21:49:03.799 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-18T21:49:03.831 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-18T21:49:03.903 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-base. 2024-04-18T21:49:03.929 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:03.979 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:04.100 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-mako. 2024-04-18T21:49:04.127 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-18T21:49:04.226 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-base. 2024-04-18T21:49:04.252 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:04.283 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T21:49:04.302 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:04.666 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-18T21:49:04.693 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-18T21:49:04.857 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-18T21:49:04.870 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-18T21:49:04.897 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-18T21:49:05.101 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cheroot. 2024-04-18T21:49:05.117 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-18T21:49:05.160 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T21:49:05.370 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-18T21:49:05.496 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-18T21:49:05.523 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-18T21:49:05.563 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-18T21:49:05.656 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-18T21:49:05.682 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-18T21:49:05.682 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-18T21:49:05.708 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-18T21:49:05.722 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-18T21:49:05.740 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-18T21:49:05.866 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-18T21:49:05.892 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-18T21:49:05.932 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-18T21:49:06.026 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webob. 2024-04-18T21:49:06.052 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-18T21:49:06.092 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-18T21:49:06.175 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cheroot. 2024-04-18T21:49:06.201 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-18T21:49:06.234 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-18T21:49:06.241 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T21:49:06.261 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-18T21:49:06.301 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-18T21:49:06.437 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-waitress. 2024-04-18T21:49:06.463 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-18T21:49:06.506 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T21:49:06.553 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-18T21:49:06.578 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-tempora. 2024-04-18T21:49:06.579 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-18T21:49:06.605 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-18T21:49:06.619 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-18T21:49:06.646 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-18T21:49:06.815 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-tempita. 2024-04-18T21:49:06.841 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-18T21:49:06.881 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T21:49:06.888 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-18T21:49:06.915 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-18T21:49:06.948 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-portend. 2024-04-18T21:49:06.955 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-18T21:49:06.975 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-18T21:49:07.015 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-18T21:49:07.159 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-paste. 2024-04-18T21:49:07.174 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-18T21:49:07.217 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T21:49:07.250 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-18T21:49:07.276 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-18T21:49:07.316 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-18T21:49:07.318 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-18T21:49:07.344 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-18T21:49:07.384 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-18T21:49:07.520 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-18T21:49:07.547 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-18T21:49:07.578 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T21:49:07.627 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-tempora. 2024-04-18T21:49:07.654 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-18T21:49:07.682 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-18T21:49:07.694 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-18T21:49:07.706 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-18T21:49:07.737 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-18T21:49:07.881 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-18T21:49:07.908 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-18T21:49:07.947 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-18T21:49:07.996 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-portend. 2024-04-18T21:49:08.023 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-18T21:49:08.071 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-18T21:49:08.091 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-natsort. 2024-04-18T21:49:08.117 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-18T21:49:08.157 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-18T21:49:08.284 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webtest. 2024-04-18T21:49:08.310 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-18T21:49:08.350 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-18T21:49:08.458 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-18T21:49:08.469 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-logutils. 2024-04-18T21:49:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-18T21:49:08.495 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-18T21:49:08.524 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-18T21:49:08.535 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-18T21:49:08.670 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pecan. 2024-04-18T21:49:08.696 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-18T21:49:08.745 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T21:49:08.796 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-18T21:49:08.823 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-18T21:49:08.827 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-18T21:49:08.853 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-18T21:49:08.863 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-18T21:49:08.902 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-18T21:49:09.083 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-certifi. 2024-04-18T21:49:09.110 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-18T21:49:09.149 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-18T21:49:09.166 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-mako. 2024-04-18T21:49:09.192 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-18T21:49:09.232 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T21:49:09.305 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-natsort. 2024-04-18T21:49:09.332 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-18T21:49:09.380 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-18T21:49:09.469 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-idna. 2024-04-18T21:49:09.487 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-18T21:49:09.527 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-18T21:49:09.577 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-18T21:49:09.604 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-18T21:49:09.643 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-18T21:49:09.725 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-logutils. 2024-04-18T21:49:09.751 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-18T21:49:09.791 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-18T21:49:09.855 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-urllib3. 2024-04-18T21:49:09.882 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-18T21:49:09.921 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T21:49:10.094 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-18T21:49:10.121 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-18T21:49:10.161 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-18T21:49:10.249 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-requests. 2024-04-18T21:49:10.276 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-18T21:49:10.316 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T21:49:10.464 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-18T21:49:10.491 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-18T21:49:10.497 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-mako. 2024-04-18T21:49:10.524 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-18T21:49:10.572 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T21:49:10.585 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-18T21:49:10.600 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-18T21:49:10.652 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T21:49:10.663 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-18T21:49:10.925 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-18T21:49:10.933 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-webob. 2024-04-18T21:49:10.952 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-18T21:49:10.959 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-18T21:49:10.963 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-18T21:49:10.979 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:10.992 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-18T21:49:10.999 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-18T21:49:11.013 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:11.270 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-18T21:49:11.296 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-18T21:49:11.327 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-waitress. 2024-04-18T21:49:11.354 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-18T21:49:11.376 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-18T21:49:11.397 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T21:49:11.403 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:11.443 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:11.444 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-18T21:49:11.697 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-webob. 2024-04-18T21:49:11.714 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-tempita. 2024-04-18T21:49:11.724 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-18T21:49:11.737 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-18T21:49:11.740 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-18T21:49:11.761 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-18T21:49:11.763 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-18T21:49:11.780 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T21:49:11.795 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-18T21:49:12.065 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-openssl. 2024-04-18T21:49:12.083 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-paste. 2024-04-18T21:49:12.091 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-18T21:49:12.109 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-18T21:49:12.124 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-waitress. 2024-04-18T21:49:12.131 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-18T21:49:12.149 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T21:49:12.151 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-18T21:49:12.195 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T21:49:12.468 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr. 2024-04-18T21:49:12.478 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-tempita. 2024-04-18T21:49:12.486 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-18T21:49:12.495 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:12.504 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-18T21:49:12.512 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-18T21:49:12.534 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:12.536 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T21:49:12.552 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T21:49:12.839 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-paste. 2024-04-18T21:49:12.863 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-18T21:49:12.865 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-18T21:49:12.890 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-18T21:49:12.905 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T21:49:12.930 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-18T21:49:12.937 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mon. 2024-04-18T21:49:12.964 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:13.004 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:13.191 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-webtest. 2024-04-18T21:49:13.207 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-18T21:49:13.233 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-18T21:49:13.241 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-18T21:49:13.248 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-18T21:49:13.283 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T21:49:13.536 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pecan. 2024-04-18T21:49:13.540 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-osd. 2024-04-18T21:49:13.552 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-18T21:49:13.562 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-18T21:49:13.568 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:13.579 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-18T21:49:13.599 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:13.602 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T21:49:13.619 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-18T21:49:13.922 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-webtest. 2024-04-18T21:49:13.948 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-18T21:49:13.973 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-certifi. 2024-04-18T21:49:13.988 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-18T21:49:14.000 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-18T21:49:14.040 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-18T21:49:14.300 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pecan. 2024-04-18T21:49:14.326 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-18T21:49:14.360 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-idna. 2024-04-18T21:49:14.366 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T21:49:14.386 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-18T21:49:14.426 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-18T21:49:14.627 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph. 2024-04-18T21:49:14.653 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../066-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:14.693 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:14.721 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-certifi. 2024-04-18T21:49:14.737 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-18T21:49:14.737 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-urllib3. 2024-04-18T21:49:14.752 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-18T21:49:14.771 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-18T21:49:14.804 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T21:49:15.029 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-fuse. 2024-04-18T21:49:15.056 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:15.074 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-idna. 2024-04-18T21:49:15.096 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:15.100 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-18T21:49:15.124 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-requests. 2024-04-18T21:49:15.140 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-18T21:49:15.150 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-18T21:49:15.190 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T21:49:15.424 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mds. 2024-04-18T21:49:15.440 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:15.459 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-18T21:49:15.460 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-urllib3. 2024-04-18T21:49:15.475 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-18T21:49:15.482 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:15.486 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-18T21:49:15.509 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T21:49:15.526 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T21:49:15.804 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-18T21:49:15.820 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:15.862 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:15.868 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package cephadm. 2024-04-18T21:49:15.879 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-requests. 2024-04-18T21:49:15.895 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../069-cephadm_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:15.906 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-18T21:49:15.934 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:15.946 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T21:49:16.234 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-18T21:49:16.240 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-18T21:49:16.254 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-18T21:49:16.260 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:16.267 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-18T21:49:16.281 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-18T21:49:16.298 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T21:49:16.300 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:16.320 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-18T21:49:16.568 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-18T21:49:16.583 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:16.611 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-18T21:49:16.626 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:16.632 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-18T21:49:16.638 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-18T21:49:16.659 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T21:49:16.678 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-18T21:49:16.698 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:16.947 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-18T21:49:16.956 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-openssl. 2024-04-18T21:49:16.963 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:16.982 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-18T21:49:16.997 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:17.014 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-18T21:49:17.317 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-18T21:49:17.325 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr. 2024-04-18T21:49:17.344 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-18T21:49:17.352 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:17.383 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-18T21:49:17.392 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:17.686 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-openssl. 2024-04-18T21:49:17.687 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-babel. 2024-04-18T21:49:17.712 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-18T21:49:17.713 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T21:49:17.752 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-18T21:49:17.753 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:17.786 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mon. 2024-04-18T21:49:17.813 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:17.869 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:18.081 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jinja2. 2024-04-18T21:49:18.081 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr. 2024-04-18T21:49:18.107 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:18.108 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-18T21:49:18.147 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T21:49:18.147 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:18.423 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-osd. 2024-04-18T21:49:18.450 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:18.489 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:18.492 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-18T21:49:18.500 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mon. 2024-04-18T21:49:18.507 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:18.526 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:18.550 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:18.566 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:18.928 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-18T21:49:18.955 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-18T21:49:18.994 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-18T21:49:19.187 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-osd. 2024-04-18T21:49:19.213 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:19.287 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:19.322 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-routes. 2024-04-18T21:49:19.349 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-18T21:49:19.389 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T21:49:19.526 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph. 2024-04-18T21:49:19.541 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../066-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:19.584 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:19.709 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-18T21:49:19.735 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:19.806 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:19.912 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-fuse. 2024-04-18T21:49:19.939 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:19.978 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:20.282 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mds. 2024-04-18T21:49:20.308 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:20.340 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph. 2024-04-18T21:49:20.348 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:20.355 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../066-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:20.398 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:20.725 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-fuse. 2024-04-18T21:49:20.752 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:20.768 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package cephadm. 2024-04-18T21:49:20.795 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../069-cephadm_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:20.817 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:20.834 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:21.120 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-18T21:49:21.147 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-18T21:49:21.147 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-18T21:49:21.163 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-18T21:49:21.178 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-18T21:49:21.187 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mds. 2024-04-18T21:49:21.197 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T21:49:21.214 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:21.254 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:21.465 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-18T21:49:21.491 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T21:49:21.523 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:21.659 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-joblib. 2024-04-18T21:49:21.686 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-18T21:49:21.725 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T21:49:21.832 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package cephadm. 2024-04-18T21:49:21.858 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../069-cephadm_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:21.898 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:22.112 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-18T21:49:22.141 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-18T21:49:22.178 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-18T21:49:22.201 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-18T21:49:22.227 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-18T21:49:22.267 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-18T21:49:22.444 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-babel. 2024-04-18T21:49:22.448 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-sklearn. 2024-04-18T21:49:22.464 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-18T21:49:22.471 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T21:49:22.502 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:22.506 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T21:49:22.603 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-18T21:49:22.620 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T21:49:22.661 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:22.797 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-jinja2. 2024-04-18T21:49:22.824 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-18T21:49:22.863 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T21:49:23.183 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-18T21:49:23.198 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-18T21:49:23.210 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:23.225 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:23.249 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:23.341 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:23.586 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-18T21:49:23.613 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-18T21:49:23.641 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-babel. 2024-04-18T21:49:23.652 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-18T21:49:23.657 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T21:49:23.691 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:23.957 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-routes. 2024-04-18T21:49:23.977 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jinja2. 2024-04-18T21:49:23.985 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-18T21:49:23.993 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-18T21:49:24.024 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T21:49:24.027 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T21:49:24.229 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cachetools. 2024-04-18T21:49:24.255 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-18T21:49:24.295 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-18T21:49:24.360 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-18T21:49:24.380 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-18T21:49:24.388 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:24.407 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:24.427 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:24.446 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:24.597 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-18T21:49:24.625 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-18T21:49:24.664 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-18T21:49:24.766 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-18T21:49:24.793 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-18T21:49:24.833 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-18T21:49:25.119 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-routes. 2024-04-18T21:49:25.145 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-18T21:49:25.185 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T21:49:25.201 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-18T21:49:25.228 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-18T21:49:25.267 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T21:49:25.530 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-18T21:49:25.557 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:25.587 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rsa. 2024-04-18T21:49:25.596 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:25.614 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-18T21:49:25.653 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-18T21:49:25.699 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-18T21:49:25.726 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-18T21:49:25.765 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T21:49:25.965 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-google-auth. 2024-04-18T21:49:25.992 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-18T21:49:26.032 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-18T21:49:26.360 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-18T21:49:26.387 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-18T21:49:26.426 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T21:49:26.427 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-joblib. 2024-04-18T21:49:26.443 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-18T21:49:26.477 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T21:49:26.729 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-websocket. 2024-04-18T21:49:26.757 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-18T21:49:26.788 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-18T21:49:26.796 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-18T21:49:26.804 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-18T21:49:26.818 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-18T21:49:26.834 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-18T21:49:26.838 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-18T21:49:26.868 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T21:49:27.116 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-18T21:49:27.116 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-sklearn. 2024-04-18T21:49:27.143 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-18T21:49:27.143 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-18T21:49:27.182 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T21:49:27.182 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T21:49:27.372 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-joblib. 2024-04-18T21:49:27.399 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-18T21:49:27.438 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T21:49:27.800 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-18T21:49:27.827 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-18T21:49:27.841 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-18T21:49:27.846 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-18T21:49:27.857 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:27.862 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:27.866 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-18T21:49:27.904 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:27.976 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:28.160 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-sklearn. 2024-04-18T21:49:28.187 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-18T21:49:28.218 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T21:49:28.224 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-18T21:49:28.251 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-18T21:49:28.282 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T21:49:28.610 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-18T21:49:28.637 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T21:49:28.676 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T21:49:28.877 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-18T21:49:28.904 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:28.913 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cachetools. 2024-04-18T21:49:28.940 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-18T21:49:28.962 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package jq. 2024-04-18T21:49:28.978 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T21:49:28.979 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-18T21:49:29.020 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-18T21:49:29.021 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:29.256 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package socat. 2024-04-18T21:49:29.272 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-18T21:49:29.307 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-18T21:49:29.315 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-18T21:49:29.334 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-18T21:49:29.374 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-18T21:49:29.609 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package xmlstarlet. 2024-04-18T21:49:29.634 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-18T21:49:29.676 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-18T21:49:29.694 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-18T21:49:29.721 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-18T21:49:29.760 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T21:49:30.022 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-test. 2024-04-18T21:49:30.024 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cachetools. 2024-04-18T21:49:30.050 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:30.051 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-18T21:49:30.071 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-rsa. 2024-04-18T21:49:30.088 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:30.090 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-18T21:49:30.098 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-18T21:49:30.138 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-18T21:49:30.417 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-google-auth. 2024-04-18T21:49:30.418 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-18T21:49:30.435 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-18T21:49:30.444 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-18T21:49:30.475 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-18T21:49:30.476 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-18T21:49:30.745 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-18T21:49:30.772 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-18T21:49:30.782 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-18T21:49:30.803 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T21:49:30.809 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-18T21:49:30.840 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T21:49:31.114 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-websocket. 2024-04-18T21:49:31.118 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-rsa. 2024-04-18T21:49:31.141 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-18T21:49:31.145 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-18T21:49:31.176 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-18T21:49:31.180 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-18T21:49:31.496 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-google-auth. 2024-04-18T21:49:31.509 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-18T21:49:31.524 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-18T21:49:31.536 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-18T21:49:31.563 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-18T21:49:31.575 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T21:49:31.883 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-18T21:49:31.910 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-18T21:49:31.949 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T21:49:32.231 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-18T21:49:32.247 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:32.289 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:32.384 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-volume. 2024-04-18T21:49:32.412 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:32.435 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-websocket. 2024-04-18T21:49:32.442 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:32.462 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-18T21:49:32.502 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-18T21:49:32.617 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-18T21:49:32.644 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-18T21:49:32.683 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T21:49:32.746 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-18T21:49:32.773 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:32.797 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-18T21:49:32.812 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:32.824 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-18T21:49:32.854 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T21:49:32.953 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-18T21:49:32.980 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T21:49:33.011 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T21:49:33.107 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-18T21:49:33.122 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-18T21:49:33.165 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T21:49:33.255 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package jq. 2024-04-18T21:49:33.271 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T21:49:33.313 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-18T21:49:33.486 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-18T21:49:33.503 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-18T21:49:33.510 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-18T21:49:33.530 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:33.544 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-18T21:49:33.568 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:33.574 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package socat. 2024-04-18T21:49:33.590 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-18T21:49:33.632 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-18T21:49:33.911 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package xmlstarlet. 2024-04-18T21:49:33.913 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-18T21:49:33.915 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package nvme-cli. 2024-04-18T21:49:33.926 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-18T21:49:33.940 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-18T21:49:33.942 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-18T21:49:33.969 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-18T21:49:33.979 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T21:49:33.981 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T21:49:34.250 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-18T21:49:34.276 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T21:49:34.307 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T21:49:34.314 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-test. 2024-04-18T21:49:34.330 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:34.373 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:34.411 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package pkg-config. 2024-04-18T21:49:34.440 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-18T21:49:34.478 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T21:49:34.568 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package jq. 2024-04-18T21:49:34.595 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T21:49:34.634 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-18T21:49:34.731 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-18T21:49:34.746 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-18T21:49:34.814 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-18T21:49:34.887 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package socat. 2024-04-18T21:49:34.910 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-18T21:49:34.953 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-18T21:49:35.234 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-18T21:49:35.257 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package xmlstarlet. 2024-04-18T21:49:35.261 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-18T21:49:35.272 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-18T21:49:35.300 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-18T21:49:35.315 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-18T21:49:35.611 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pastescript. 2024-04-18T21:49:35.637 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-test. 2024-04-18T21:49:35.639 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-18T21:49:35.652 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:35.677 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-18T21:49:35.694 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:35.972 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pluggy. 2024-04-18T21:49:35.986 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-18T21:49:36.022 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-18T21:49:36.342 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-psutil. 2024-04-18T21:49:36.369 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-18T21:49:36.408 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-18T21:49:36.745 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-py. 2024-04-18T21:49:36.772 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-18T21:49:36.811 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-18T21:49:36.852 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-volume. 2024-04-18T21:49:36.880 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:36.918 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:37.106 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pygments. 2024-04-18T21:49:37.122 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-18T21:49:37.156 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T21:49:37.222 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-18T21:49:37.250 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:37.289 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:37.600 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-18T21:49:37.610 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-18T21:49:37.627 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-18T21:49:37.638 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-18T21:49:37.666 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T21:49:37.676 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-18T21:49:37.945 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-18T21:49:37.961 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-18T21:49:37.982 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-volume. 2024-04-18T21:49:37.987 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-toml. 2024-04-18T21:49:37.995 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-18T21:49:38.003 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-18T21:49:38.009 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-18T21:49:38.046 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-18T21:49:38.048 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:38.343 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-18T21:49:38.349 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pytest. 2024-04-18T21:49:38.358 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package nvme-cli. 2024-04-18T21:49:38.360 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:38.376 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-18T21:49:38.385 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-18T21:49:38.402 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:38.415 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T21:49:38.424 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T21:49:38.739 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-18T21:49:38.766 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-18T21:49:38.768 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-simplejson. 2024-04-18T21:49:38.796 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-18T21:49:38.812 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T21:49:38.846 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package pkg-config. 2024-04-18T21:49:38.870 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-18T21:49:38.873 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-18T21:49:38.912 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T21:49:39.165 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-18T21:49:39.173 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-18T21:49:39.181 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-18T21:49:39.189 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-18T21:49:39.200 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-18T21:49:39.215 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-18T21:49:39.228 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-18T21:49:39.231 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-18T21:49:39.275 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-18T21:49:39.626 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-18T21:49:39.654 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-18T21:49:39.662 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package nvme-cli. 2024-04-18T21:49:39.690 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-18T21:49:39.693 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-18T21:49:39.728 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T21:49:40.001 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package radosgw. 2024-04-18T21:49:40.012 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pastescript. 2024-04-18T21:49:40.029 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../116-radosgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:40.040 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-18T21:49:40.067 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:40.078 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-18T21:49:40.151 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package pkg-config. 2024-04-18T21:49:40.179 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-18T21:49:40.217 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T21:49:40.382 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pluggy. 2024-04-18T21:49:40.410 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-18T21:49:40.440 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-18T21:49:40.462 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-18T21:49:40.489 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-18T21:49:40.528 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-18T21:49:40.735 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-psutil. 2024-04-18T21:49:40.763 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-18T21:49:40.801 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-18T21:49:40.971 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package rbd-fuse. 2024-04-18T21:49:40.972 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-18T21:49:40.999 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:41.000 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-18T21:49:41.037 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:41.047 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-18T21:49:41.129 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-py. 2024-04-18T21:49:41.157 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-18T21:49:41.195 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-18T21:49:41.308 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pastescript. 2024-04-18T21:49:41.325 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-18T21:49:41.333 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-18T21:49:41.358 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-18T21:49:41.533 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pygments. 2024-04-18T21:49:41.560 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-18T21:49:41.575 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T21:49:41.599 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T21:49:41.661 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pluggy. 2024-04-18T21:49:41.684 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T21:49:41.689 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-18T21:49:41.728 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-18T21:49:41.971 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-18T21:49:42.039 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-psutil. 2024-04-18T21:49:42.053 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-18T21:49:42.067 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-18T21:49:42.081 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-18T21:49:42.106 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-18T21:49:42.119 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-18T21:49:42.289 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-18T21:49:42.405 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-toml. 2024-04-18T21:49:42.433 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-18T21:49:42.442 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-py. 2024-04-18T21:49:42.469 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-18T21:49:42.471 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-18T21:49:42.508 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-18T21:49:42.716 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pytest. 2024-04-18T21:49:42.722 INFO:teuthology.orchestra.run.smithi104.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-18T21:49:42.741 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-18T21:49:42.783 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T21:49:42.786 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:43.021 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pygments. 2024-04-18T21:49:43.047 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-18T21:49:43.086 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T21:49:43.094 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-simplejson. 2024-04-18T21:49:43.123 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-18T21:49:43.161 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-18T21:49:43.327 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user cephadm....done 2024-04-18T21:49:43.438 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-18T21:49:43.454 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-18T21:49:43.483 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T21:49:43.488 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-18T21:49:43.507 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-18T21:49:43.535 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-18T21:49:43.565 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-18T21:49:43.751 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-18T21:49:43.851 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-toml. 2024-04-18T21:49:43.879 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-18T21:49:43.917 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-18T21:49:44.019 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-18T21:49:44.145 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-18T21:49:44.201 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package radosgw. 2024-04-18T21:49:44.220 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pytest. 2024-04-18T21:49:44.228 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../116-radosgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:44.249 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-18T21:49:44.267 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T21:49:44.396 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-18T21:49:44.590 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-simplejson. 2024-04-18T21:49:44.617 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-18T21:49:44.657 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-18T21:49:44.672 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T21:49:44.815 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-18T21:49:44.909 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-18T21:49:44.925 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-18T21:49:44.967 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-18T21:49:45.053 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package rbd-fuse. 2024-04-18T21:49:45.069 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:45.111 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:45.133 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T21:49:45.390 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-18T21:49:45.485 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-18T21:49:45.641 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T21:49:45.738 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package radosgw. 2024-04-18T21:49:45.752 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-18T21:49:45.758 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T21:49:45.766 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../116-radosgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:45.804 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:45.995 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-18T21:49:46.020 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-18T21:49:46.299 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-18T21:49:46.322 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-18T21:49:46.572 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-18T21:49:46.632 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package rbd-fuse. 2024-04-18T21:49:46.648 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-18T21:49:46.690 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:46.716 INFO:teuthology.orchestra.run.smithi183.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-18T21:49:46.777 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:46.832 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-18T21:49:46.995 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-18T21:49:47.083 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:47.245 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T21:49:47.304 INFO:teuthology.orchestra.run.smithi183.stdout:Adding system user cephadm....done 2024-04-18T21:49:47.334 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T21:49:47.346 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T21:49:47.458 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T21:49:47.476 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T21:49:47.599 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-18T21:49:47.602 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T21:49:47.726 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-18T21:49:47.728 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-18T21:49:47.911 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-18T21:49:47.960 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-18T21:49:47.979 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-18T21:49:48.086 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-18T21:49:48.337 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-18T21:49:48.357 INFO:teuthology.orchestra.run.smithi097.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-18T21:49:48.448 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:48.468 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-18T21:49:48.494 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-18T21:49:48.557 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T21:49:48.597 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T21:49:48.683 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-18T21:49:48.731 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-18T21:49:48.964 INFO:teuthology.orchestra.run.smithi097.stdout:Adding system user cephadm....done 2024-04-18T21:49:49.043 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:49.083 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T21:49:49.087 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T21:49:49.168 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-18T21:49:49.388 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-18T21:49:49.428 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T21:49:49.451 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-18T21:49:49.640 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-18T21:49:49.686 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-18T21:49:49.687 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-18T21:49:49.766 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-18T21:49:49.945 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-18T21:49:49.971 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up zip (3.0-12build2) ... 2024-04-18T21:49:50.000 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-18T21:49:50.098 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T21:49:50.197 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-18T21:49:50.235 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T21:49:50.352 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-18T21:49:50.473 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-18T21:49:50.662 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T21:49:50.733 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-18T21:49:50.741 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T21:49:50.975 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:51.000 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T21:49:51.022 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-18T21:49:51.126 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-18T21:49:51.235 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T21:49:51.252 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T21:49:51.290 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-18T21:49:51.344 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T21:49:51.470 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T21:49:51.596 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-18T21:49:51.663 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-18T21:49:51.700 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-18T21:49:51.897 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-18T21:49:51.968 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-18T21:49:52.196 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-18T21:49:52.219 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-18T21:49:52.323 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-18T21:49:52.363 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-18T21:49:52.433 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T21:49:52.455 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T21:49:52.470 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-18T21:49:52.535 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-18T21:49:52.721 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:52.832 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T21:49:52.886 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:52.980 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T21:49:53.012 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-18T21:49:53.115 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T21:49:53.116 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T21:49:53.241 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T21:49:53.271 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T21:49:53.367 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-18T21:49:53.468 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-18T21:49:53.539 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-18T21:49:53.609 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-18T21:49:53.735 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-18T21:49:53.849 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up zip (3.0-12build2) ... 2024-04-18T21:49:53.870 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:53.959 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-18T21:49:53.966 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T21:49:53.983 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-18T21:49:54.046 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T21:49:54.172 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-18T21:49:54.196 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-idna (3.3-1) ... 2024-04-18T21:49:54.522 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T21:49:54.531 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T21:49:54.618 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T21:49:54.657 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-18T21:49:54.852 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T21:49:54.916 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T21:49:54.962 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-18T21:49:55.088 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T21:49:55.159 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-18T21:49:55.381 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-18T21:49:55.436 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up zip (3.0-12build2) ... 2024-04-18T21:49:55.561 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T21:49:55.677 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T21:49:55.773 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-18T21:49:55.946 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:49:56.016 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T21:49:56.080 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-18T21:49:56.197 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T21:49:56.347 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T21:49:56.368 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T21:49:56.464 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T21:49:56.465 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-18T21:49:56.566 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T21:49:56.590 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-18T21:49:56.635 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T21:49:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T21:49:56.859 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-18T21:49:56.979 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-18T21:49:57.009 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-18T21:49:57.110 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-18T21:49:57.230 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-18T21:49:57.361 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:49:57.372 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:57.377 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-18T21:49:57.461 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-18T21:49:57.611 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T21:49:57.682 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-idna (3.3-1) ... 2024-04-18T21:49:57.755 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-18T21:49:58.006 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-18T21:49:58.008 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T21:49:58.204 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T21:49:58.256 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-18T21:49:58.499 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-18T21:49:58.514 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T21:49:58.750 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-18T21:49:58.874 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-18T21:49:59.018 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T21:49:59.141 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-18T21:49:59.152 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-18T21:49:59.205 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T21:49:59.242 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:49:59.436 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T21:49:59.457 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:49:59.510 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-idna (3.3-1) ... 2024-04-18T21:49:59.562 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T21:49:59.591 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-18T21:49:59.795 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T21:49:59.834 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T21:49:59.855 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-18T21:49:59.935 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-18T21:50:00.044 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T21:50:00.156 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-18T21:50:00.362 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-18T21:50:00.407 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T21:50:00.533 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-18T21:50:00.630 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-18T21:50:00.659 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T21:50:00.749 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-18T21:50:00.889 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:50:00.967 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T21:50:00.986 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T21:50:01.015 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-18T21:50:01.103 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T21:50:01.193 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:50:01.325 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-18T21:50:01.327 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-18T21:50:01.538 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-18T21:50:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-18T21:50:01.570 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T21:50:01.696 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-18T21:50:01.789 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-18T21:50:01.810 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-18T21:50:01.822 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T21:50:01.898 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-18T21:50:02.045 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-18T21:50:02.115 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-18T21:50:02.141 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:50:02.267 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T21:50:02.296 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-18T21:50:02.366 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-18T21:50:02.598 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T21:50:02.600 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:50:02.601 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T21:50:02.727 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-18T21:50:02.732 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-18T21:50:02.919 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T21:50:03.012 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-18T21:50:03.017 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T21:50:03.126 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T21:50:03.195 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-18T21:50:03.245 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-18T21:50:03.321 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-18T21:50:03.402 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-18T21:50:03.505 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-18T21:50:03.580 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-18T21:50:03.687 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-18T21:50:03.789 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-18T21:50:03.848 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T21:50:03.946 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T21:50:04.041 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-18T21:50:04.072 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-18T21:50:04.198 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T21:50:04.201 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-18T21:50:04.292 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T21:50:04.304 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-18T21:50:04.418 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-18T21:50:04.541 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T21:50:04.611 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-18T21:50:04.651 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T21:50:04.694 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T21:50:04.820 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T21:50:04.896 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T21:50:05.036 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-18T21:50:05.112 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-18T21:50:05.223 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T21:50:05.295 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-18T21:50:05.349 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:05.421 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-18T21:50:05.430 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-18T21:50:05.475 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:05.601 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-18T21:50:05.680 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:50:05.689 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T21:50:05.816 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-18T21:50:05.832 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T21:50:05.925 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T21:50:05.937 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:05.996 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-18T21:50:06.055 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:06.149 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T21:50:06.181 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-18T21:50:06.235 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T21:50:06.344 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T21:50:06.450 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T21:50:06.457 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:06.583 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:06.709 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:06.710 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-18T21:50:06.720 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-18T21:50:06.811 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-18T21:50:06.920 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T21:50:06.920 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T21:50:06.972 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-18T21:50:07.097 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-18T21:50:07.357 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T21:50:07.362 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-18T21:50:07.436 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:07.483 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T21:50:07.570 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:07.647 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T21:50:07.730 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:07.792 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T21:50:07.848 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T21:50:07.991 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-18T21:50:08.076 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T21:50:08.343 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-18T21:50:08.369 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-18T21:50:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-18T21:50:08.611 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T21:50:08.763 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-18T21:50:08.955 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T21:50:09.064 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T21:50:09.067 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.168 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.269 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-18T21:50:09.310 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.426 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-18T21:50:09.435 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.561 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.588 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.687 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.714 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.805 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:09.811 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-18T21:50:09.839 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-18T21:50:10.087 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T21:50:10.124 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:10.242 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:10.343 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:10.348 INFO:teuthology.orchestra.run.smithi104.stdout:Adding group ceph....done 2024-04-18T21:50:10.439 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T21:50:10.538 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T21:50:10.539 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T21:50:10.566 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:10.692 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:10.715 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user ceph....done 2024-04-18T21:50:10.817 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-18T21:50:10.883 INFO:teuthology.orchestra.run.smithi104.stdout:Setting system user ceph properties....done 2024-04-18T21:50:10.902 INFO:teuthology.orchestra.run.smithi104.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-18T21:50:10.969 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.038 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-18T21:50:11.095 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.104 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.213 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.221 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.339 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-18T21:50:11.372 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T21:50:11.387 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-18T21:50:11.624 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.725 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.826 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.839 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:11.957 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:12.029 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T21:50:12.029 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T21:50:12.527 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:12.567 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:12.582 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T21:50:12.583 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T21:50:12.653 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:12.670 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:12.777 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:12.779 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:12.905 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T21:50:12.911 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:13.038 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:13.076 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:13.541 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-18T21:50:13.589 INFO:teuthology.orchestra.run.smithi183.stdout:Adding group ceph....done 2024-04-18T21:50:13.941 INFO:teuthology.orchestra.run.smithi183.stdout:Adding system user ceph....done 2024-04-18T21:50:13.994 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:14.101 INFO:teuthology.orchestra.run.smithi183.stdout:Setting system user ceph properties....done 2024-04-18T21:50:14.109 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:14.119 INFO:teuthology.orchestra.run.smithi183.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-18T21:50:14.212 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T21:50:14.212 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T21:50:14.235 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:14.253 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-18T21:50:14.344 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:14.445 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:14.546 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:14.555 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-18T21:50:14.654 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:14.868 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T21:50:14.868 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T21:50:15.013 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:15.063 INFO:teuthology.orchestra.run.smithi097.stdout:Adding group ceph....done 2024-04-18T21:50:15.139 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:15.313 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:15.448 INFO:teuthology.orchestra.run.smithi097.stdout:Adding system user ceph....done 2024-04-18T21:50:15.584 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T21:50:15.585 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T21:50:15.592 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T21:50:15.592 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T21:50:15.617 INFO:teuthology.orchestra.run.smithi097.stdout:Setting system user ceph properties....done 2024-04-18T21:50:15.632 INFO:teuthology.orchestra.run.smithi097.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-18T21:50:15.773 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-18T21:50:16.025 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:16.031 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:16.113 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-18T21:50:16.157 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:16.308 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:16.373 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-18T21:50:16.546 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T21:50:16.546 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T21:50:16.613 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:16.739 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:16.810 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:17.002 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:17.046 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T21:50:17.046 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T21:50:17.145 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:17.250 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T21:50:17.250 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T21:50:17.262 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:17.405 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:17.555 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:17.718 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:17.766 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-18T21:50:17.819 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T21:50:17.819 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T21:50:17.875 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-18T21:50:18.028 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-18T21:50:18.041 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-18T21:50:18.271 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:18.519 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:18.609 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T21:50:18.609 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T21:50:18.738 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T21:50:18.738 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T21:50:18.760 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-18T21:50:19.090 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:19.221 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:19.224 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:19.350 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:19.437 INFO:teuthology.orchestra.run.smithi104.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-18T21:50:19.439 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T21:50:19.439 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T21:50:19.443 DEBUG:teuthology.parallel:result is None 2024-04-18T21:50:19.547 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T21:50:19.547 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T21:50:19.931 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:20.051 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:20.202 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:20.211 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T21:50:20.211 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T21:50:20.328 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:20.479 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:20.691 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:20.824 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-18T21:50:20.825 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:20.924 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-18T21:50:20.976 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:21.068 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-18T21:50:21.187 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T21:50:21.187 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T21:50:21.686 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:21.759 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-18T21:50:21.803 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:21.904 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:22.064 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-18T21:50:22.388 INFO:teuthology.orchestra.run.smithi183.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-18T21:50:22.393 DEBUG:teuthology.parallel:result is None 2024-04-18T21:50:22.408 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-18T21:50:22.517 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-18T21:50:22.661 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-18T21:50:23.720 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-18T21:50:24.419 INFO:teuthology.orchestra.run.smithi097.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-18T21:50:24.424 DEBUG:teuthology.parallel:result is None 2024-04-18T21:50:24.424 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-18T21:50:24.640 DEBUG:teuthology.orchestra.run.smithi097:> dpkg-query -W -f '${Version}' ceph 2024-04-18T21:50:24.659 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:50:24.659 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:50:24.660 INFO:teuthology.task.install:The correct ceph version 19.0.0-2369-g1a6a3e74-1jammy is installed. 2024-04-18T21:50:24.661 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-18T21:50:24.875 DEBUG:teuthology.orchestra.run.smithi104:> dpkg-query -W -f '${Version}' ceph 2024-04-18T21:50:24.895 INFO:teuthology.orchestra.run.smithi104.stdout:19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:50:24.895 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:50:24.895 INFO:teuthology.task.install:The correct ceph version 19.0.0-2369-g1a6a3e74-1jammy is installed. 2024-04-18T21:50:24.897 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-18T21:50:25.036 DEBUG:teuthology.orchestra.run.smithi183:> dpkg-query -W -f '${Version}' ceph 2024-04-18T21:50:25.055 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:50:25.055 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-18T21:50:25.055 INFO:teuthology.task.install:The correct ceph version 19.0.0-2369-g1a6a3e74-1jammy is installed. 2024-04-18T21:50:25.057 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-18T21:50:25.057 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:25.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-18T21:50:25.071 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:50:25.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-18T21:50:25.085 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:50:25.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-18T21:50:25.112 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-18T21:50:25.112 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:25.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-04-18T21:50:25.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-18T21:50:25.181 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:50:25.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-04-18T21:50:25.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-18T21:50:25.247 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:50:25.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-04-18T21:50:25.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-18T21:50:25.314 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-18T21:50:25.314 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:25.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-18T21:50:25.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-18T21:50:25.381 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:50:25.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-18T21:50:25.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-18T21:50:25.448 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:50:25.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-18T21:50:25.461 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-18T21:50:25.514 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-18T21:50:25.514 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:25.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-04-18T21:50:25.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-18T21:50:25.585 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:50:25.585 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-04-18T21:50:25.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-18T21:50:25.651 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:50:25.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-04-18T21:50:25.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-18T21:50:25.718 INFO:teuthology.run_tasks:Running task ceph... 2024-04-18T21:50:25.798 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-18T21:50:25.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /var/log/ceph 2024-04-18T21:50:25.801 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /var/log/ceph 2024-04-18T21:50:25.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2024-04-18T21:50:25.816 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-18T21:50:25.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-18T21:50:25.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-18T21:50:25.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-18T21:50:25.869 INFO:tasks.ceph:Creating extra log directories... 2024-04-18T21:50:25.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-18T21:50:25.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-18T21:50:25.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-18T21:50:25.936 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-18T21:50:25.936 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-18T21:50:25.937 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549', 'branch': 'squid', 'description': 'rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7662549', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd0bf63dd179c05751e4be49477389aa290c73089'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'], ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'], ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']], 'seed': 6512, 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32', 'sleep_before_teardown': 0, 'subset': '28391/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d0bf63dd179c05751e4be49477389aa290c73089/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd0bf63dd179c05751e4be49477389aa290c73089', 'targets': {'smithi097.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8BTLlHsQIqVUe8iE//YUlOuJdavNucK3J/5KjI/Uys9To7LIC1sYGPUZgtrccolcjJ/hhbDNlQ+CdWV45drZE=', 'smithi104.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMvqbcsyJ/fgp+yQJNDvEpdvtoJWniApzcFsC+8Ew8QoTjwm+ZtaV4Mgm9122E3dwtsqQeP68oLn/q2qpYAlPNA=', 'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNt/faOsP8WklJZIcxe6MukZBvmG9a554VAh3KDl1EZAoFU77pNllWrJYYQEqlebE6ssaNI90pAxMYpoX94csXw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 4, 'l': 3, 'm': 2, 'name': 'lrcprofile', 'plugin': 'lrc'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-18_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-18T21:50:25.937 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-18T21:50:25.967 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-18T21:50:25.970 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-18T21:50:25.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/run/ceph 2024-04-18T21:50:26.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/run/ceph 2024-04-18T21:50:26.018 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2024-04-18T21:50:26.029 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:26.030 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-04-18T21:50:26.074 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-18T21:50:26.074 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-04-18T21:50:26.123 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-18T21:50:26.123 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:26.123 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-04-18T21:50:26.123 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:26.123 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-18 21:45:05.530137590 +0000 2024-04-18T21:50:26.123 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-18 21:45:05.274139874 +0000 2024-04-18T21:50:26.123 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-18 21:45:05.274139874 +0000 2024-04-18T21:50:26.123 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2024-04-18T21:50:26.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-18T21:50:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-18T21:50:26.177 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-18T21:50:26.177 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000408259 s, 1.3 MB/s 2024-04-18T21:50:26.178 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-18T21:50:26.227 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-04-18T21:50:26.278 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-18T21:50:26.278 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:26.279 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-18T21:50:26.279 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:26.279 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-18 21:45:05.990133486 +0000 2024-04-18T21:50:26.279 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-18 21:45:05.738135735 +0000 2024-04-18T21:50:26.279 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-18 21:45:05.738135735 +0000 2024-04-18T21:50:26.279 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2024-04-18T21:50:26.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-18T21:50:26.332 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-18T21:50:26.332 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-18T21:50:26.332 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000362723 s, 1.4 MB/s 2024-04-18T21:50:26.333 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-18T21:50:26.383 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-04-18T21:50:26.430 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-18T21:50:26.430 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:26.430 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-04-18T21:50:26.430 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:26.430 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-18 21:45:06.478129133 +0000 2024-04-18T21:50:26.431 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-18 21:45:06.222131417 +0000 2024-04-18T21:50:26.431 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-18 21:45:06.222131417 +0000 2024-04-18T21:50:26.431 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2024-04-18T21:50:26.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-18T21:50:26.484 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-18T21:50:26.484 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-18T21:50:26.484 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000402385 s, 1.3 MB/s 2024-04-18T21:50:26.485 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-18T21:50:26.539 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-04-18T21:50:26.586 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-18T21:50:26.586 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:26.586 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-04-18T21:50:26.586 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:26.586 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-18 21:45:06.950124923 +0000 2024-04-18T21:50:26.586 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-18 21:45:06.710127064 +0000 2024-04-18T21:50:26.586 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-18 21:45:06.710127064 +0000 2024-04-18T21:50:26.586 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2024-04-18T21:50:26.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-18T21:50:26.640 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-18T21:50:26.640 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-18T21:50:26.640 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000359909 s, 1.4 MB/s 2024-04-18T21:50:26.642 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-18T21:50:26.691 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:26.692 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:50:26.692 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-04-18T21:50:26.698 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-18T21:50:26.698 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-04-18T21:50:26.745 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-18T21:50:26.745 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:26.745 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-04-18T21:50:26.745 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:26.745 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-18 21:45:05.283089343 +0000 2024-04-18T21:50:26.745 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-18 21:45:05.279089379 +0000 2024-04-18T21:50:26.745 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-18 21:45:05.279089379 +0000 2024-04-18T21:50:26.746 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-04-18T21:50:26.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-18T21:50:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-18T21:50:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-18T21:50:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000411686 s, 1.2 MB/s 2024-04-18T21:50:26.801 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-18T21:50:26.850 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-04-18T21:50:26.897 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-18T21:50:26.897 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:26.897 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 895 Links: 1 2024-04-18T21:50:26.897 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:26.897 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-18 21:45:06.067082340 +0000 2024-04-18T21:50:26.897 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-18 21:45:05.791084805 +0000 2024-04-18T21:50:26.897 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-18 21:45:05.791084805 +0000 2024-04-18T21:50:26.897 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-04-18T21:50:26.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-18T21:50:26.951 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-18T21:50:26.951 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-18T21:50:26.951 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000419553 s, 1.2 MB/s 2024-04-18T21:50:26.952 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-18T21:50:27.002 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-04-18T21:50:27.048 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-18T21:50:27.049 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:27.049 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-04-18T21:50:27.049 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:27.049 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-18 21:45:06.595077623 +0000 2024-04-18T21:50:27.049 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-18 21:45:06.323080054 +0000 2024-04-18T21:50:27.049 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-18 21:45:06.323080054 +0000 2024-04-18T21:50:27.049 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-04-18T21:50:27.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-18T21:50:27.102 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-18T21:50:27.102 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-18T21:50:27.102 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000346244 s, 1.5 MB/s 2024-04-18T21:50:27.103 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-18T21:50:27.150 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-04-18T21:50:27.196 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-18T21:50:27.197 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:27.197 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-04-18T21:50:27.197 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:27.197 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-18 21:45:07.087073229 +0000 2024-04-18T21:50:27.197 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-18 21:45:06.823075588 +0000 2024-04-18T21:50:27.197 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-18 21:45:06.823075588 +0000 2024-04-18T21:50:27.197 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-04-18T21:50:27.197 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-18T21:50:27.250 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-18T21:50:27.250 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-18T21:50:27.250 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000276631 s, 1.9 MB/s 2024-04-18T21:50:27.251 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-18T21:50:27.298 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:27.298 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:50:27.298 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-04-18T21:50:27.304 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-18T21:50:27.304 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-04-18T21:50:27.351 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-18T21:50:27.352 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:27.352 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 876 Links: 1 2024-04-18T21:50:27.352 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:27.352 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-04-18 21:45:07.422840855 +0000 2024-04-18T21:50:27.352 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-04-18 21:45:07.158843211 +0000 2024-04-18T21:50:27.352 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-04-18 21:45:07.158843211 +0000 2024-04-18T21:50:27.352 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2024-04-18T21:50:27.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-18T21:50:27.406 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-04-18T21:50:27.406 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-04-18T21:50:27.406 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000447569 s, 1.1 MB/s 2024-04-18T21:50:27.407 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-18T21:50:27.456 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-04-18T21:50:27.503 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-18T21:50:27.503 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:27.503 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-04-18T21:50:27.503 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:27.503 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-04-18 21:45:07.650838822 +0000 2024-04-18T21:50:27.503 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-04-18 21:45:07.646838858 +0000 2024-04-18T21:50:27.503 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-04-18 21:45:07.646838858 +0000 2024-04-18T21:50:27.503 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2024-04-18T21:50:27.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-18T21:50:27.557 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-04-18T21:50:27.557 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-04-18T21:50:27.557 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000383063 s, 1.3 MB/s 2024-04-18T21:50:27.558 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-18T21:50:27.608 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-04-18T21:50:27.655 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-18T21:50:27.655 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:27.655 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-04-18T21:50:27.655 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:27.655 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-04-18 21:45:08.398832149 +0000 2024-04-18T21:50:27.655 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-04-18 21:45:08.158834290 +0000 2024-04-18T21:50:27.655 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-04-18 21:45:08.158834290 +0000 2024-04-18T21:50:27.655 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2024-04-18T21:50:27.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-18T21:50:27.709 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-04-18T21:50:27.709 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-04-18T21:50:27.709 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000408862 s, 1.3 MB/s 2024-04-18T21:50:27.710 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-18T21:50:27.760 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-04-18T21:50:27.807 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-18T21:50:27.807 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T21:50:27.807 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 924 Links: 1 2024-04-18T21:50:27.807 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T21:50:27.807 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-04-18 21:45:08.890827760 +0000 2024-04-18T21:50:27.807 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-04-18 21:45:08.634830043 +0000 2024-04-18T21:50:27.807 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-04-18 21:45:08.634830043 +0000 2024-04-18T21:50:27.807 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2024-04-18T21:50:27.807 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-18T21:50:27.861 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-04-18T21:50:27.861 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-04-18T21:50:27.861 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00038671 s, 1.3 MB/s 2024-04-18T21:50:27.862 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-18T21:50:27.912 INFO:tasks.ceph:osd dev map: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:27.912 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'}} 2024-04-18T21:50:27.912 INFO:tasks.ceph:Generating config... 2024-04-18T21:50:27.913 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-18T21:50:27.913 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-18T21:50:27.913 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-18T21:50:27.914 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd max backfills = 6 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-18T21:50:27.915 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-18T21:50:27.915 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-18T21:50:27.916 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2024-04-18T21:50:27.916 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2024-04-18T21:50:27.916 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-04-18T21:50:27.916 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-04-18T21:50:27.916 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-18T21:50:27.916 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-18T21:50:27.916 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-18T21:50:27.916 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-18T21:50:27.916 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-18T21:50:27.916 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-18T21:50:27.916 INFO:tasks.ceph:Setting up mon.a... 2024-04-18T21:50:27.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-18T21:50:27.950 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.keyring 2024-04-18T21:50:27.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-18T21:50:28.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-18T21:50:28.104 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.97'), ('mon.b', '172.21.15.104'), ('mon.c', '172.21.15.183')] 2024-04-18T21:50:28.104 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.97,172.21.15.104,172.21.15.183', 'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-04-18T21:50:28.105 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:28.105 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-18T21:50:28.151 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.97 --add b 172.21.15.104 --add c 172.21.15.183 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:28.231 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:28.231 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: generated fsid 9a7b119f-53a7-4e46-8d6e-229eac84ebee 2024-04-18T21:50:28.231 INFO:teuthology.orchestra.run.smithi097.stdout:setting min_mon_release = quincy 2024-04-18T21:50:28.231 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 0 2024-04-18T21:50:28.231 INFO:teuthology.orchestra.run.smithi097.stdout:fsid 9a7b119f-53a7-4e46-8d6e-229eac84ebee 2024-04-18T21:50:28.231 INFO:teuthology.orchestra.run.smithi097.stdout:last_changed 2024-04-18T21:50:28.231565+0000 2024-04-18T21:50:28.232 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-04-18T21:50:28.231565+0000 2024-04-18T21:50:28.232 INFO:teuthology.orchestra.run.smithi097.stdout:min_mon_release 17 (quincy) 2024-04-18T21:50:28.232 INFO:teuthology.orchestra.run.smithi097.stdout:election_strategy: 1 2024-04-18T21:50:28.232 INFO:teuthology.orchestra.run.smithi097.stdout:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-18T21:50:28.232 INFO:teuthology.orchestra.run.smithi097.stdout:1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-18T21:50:28.232 INFO:teuthology.orchestra.run.smithi097.stdout:2: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.c 2024-04-18T21:50:28.232 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-18T21:50:28.233 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-18T21:50:28.278 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9a7b119f-53a7-4e46-8d6e-229eac84ebee... 2024-04-18T21:50:28.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-18T21:50:28.323 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-18T21:50:28.326 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-18T21:50:28.351 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-04-18T21:50:28.351 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.352 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-04-18T21:50:28.353 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-04-18T21:50:28.354 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,172.21.15.104,172.21.15.183" 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: mon election default strategy = 1 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: osd_async_recovery_min_cost = 1 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 9a7b119f-53a7-4e46-8d6e-229eac84ebee 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.355 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 120 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-04-18T21:50:28.356 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-04-18T21:50:28.357 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: debug monc = 20 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = stupid 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore zero block detection = True 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 20 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 20 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-04-18T21:50:28.358 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 10 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd blocked scrub grace period = 3600 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug reject backfill probability = 0.3 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify cached snaps = True 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify missing on start = True 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd max backfills = 6 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd max markdown count = 1000 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock override recovery settings = True 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd op queue = debug_random 2024-04-18T21:50:28.359 INFO:teuthology.orchestra.run.smithi097.stdout: osd op queue cut off = debug_random 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub min interval = 60 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-04-18T21:50:28.360 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-04-18T21:50:28.361 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: mon scrub interval = 300 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-04-18T21:50:28.362 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-18T21:50:28.363 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-18T21:50:28.363 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-04-18T21:50:28.363 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-04-18T21:50:28.363 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-04-18T21:50:28.364 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-04-18T21:50:28.365 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-04-18T21:50:28.366 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.97,172.21.15.104,172.21.15.183" 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: mon election default strategy = 1 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: osd_async_recovery_min_cost = 1 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 9a7b119f-53a7-4e46-8d6e-229eac84ebee 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 120 2024-04-18T21:50:28.367 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-04-18T21:50:28.368 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-04-18T21:50:28.369 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = stupid 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore zero block detection = True 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 20 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 20 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 10 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: osd blocked scrub grace period = 3600 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug reject backfill probability = 0.3 2024-04-18T21:50:28.370 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify cached snaps = True 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify missing on start = True 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd max backfills = 6 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd max markdown count = 1000 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock override recovery settings = True 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue = debug_random 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue cut off = debug_random 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub min interval = 60 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.371 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-04-18T21:50:28.372 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-04-18T21:50:28.373 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: mon scrub interval = 300 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-04-18T21:50:28.374 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-04-18T21:50:28.375 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2024-04-18T21:50:28.375 INFO:teuthology.orchestra.run.smithi104.stdout: chdir = "" 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: auth supported = cephx 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: filestore xattr use omap = true 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: mon clock drift allowed = 1.000 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: osd crush chooseleaf type = 0 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: auth debug = true 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on old message = true 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on bug = true 2024-04-18T21:50:28.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon pg warn max object skew = 0 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: # disable pg_autoscaler by default for new pools 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default size = 2 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow primary affinity = true 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow pg remap = true 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on legacy crush tunables = false 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on crush straw calc version zero = false 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on no sortbitwise = false 2024-04-18T21:50:28.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on osd down out interval zero = false 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on too few osds = false 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon_allow_pool_size_one = true 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: osd default data pool replay window = 5 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon allow pool delete = true 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon cluster log file level = debug 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: debug asserts on shutdown = true 2024-04-18T21:50:28.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon health detail to clog = false 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: mon host = "172.21.15.97,172.21.15.104,172.21.15.183" 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: mon election default strategy = 1 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd_async_recovery_min_cost = 1 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = 9a7b119f-53a7-4e46-8d6e-229eac84ebee 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout:[osd] 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd journal size = 100 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub load threshold = 5.0 2024-04-18T21:50:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub max interval = 120 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock profile = high_recovery_ops 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd recover clone overlap = true 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd recovery max chunk = 1048576 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug shutdown = true 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug op order = true 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify stray on activate = true 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug trim objects = true 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd open classes on start = true 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug pg log writeout = true 2024-04-18T21:50:28.380 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: osd deep scrub update digest min age = 30 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: osd map max advance = 10 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: journal zero on create = true 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: filestore ondisk finisher threads = 3 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: filestore apply finisher threads = 3 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: bdev debug aio = true 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug misdirected ops = true 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 20 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: bdev async discard = True 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: bdev enable discard = True 2024-04-18T21:50:28.381 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore allocator = stupid 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore block size = 96636764160 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore fsck on mount = True 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore zero block detection = True 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluefs = 20 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluestore = 20 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug osd = 20 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug rocksdb = 10 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd backfillfull_ratio = 0.85 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd full ratio = 0.9 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd nearfull ratio = 0.8 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd blocked scrub grace period = 3600 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug reject backfill probability = 0.3 2024-04-18T21:50:28.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify cached snaps = True 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify missing on start = True 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd failsafe full ratio = 0.95 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd max backfills = 6 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd max markdown count = 1000 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock override recovery settings = True 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd objectstore = bluestore 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue = debug_random 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue cut off = debug_random 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub min interval = 60 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr] 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: debug mgr = 20 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-04-18T21:50:28.383 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: mgr/telemetry/nag = false 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout:[mon] 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: debug paxos = 20 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon data avail warn = 5 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon mgr mkfs grace = 240 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-04-18T21:50:28.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd reporter subtree level = osd 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd prime pg temp = true 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: auth mon ticket ttl = 660 # 11m 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: auth service ticket ttl = 240 # 4m 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: # don't complain about insecure global_id in the test suite 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: # 1m isn't quite enough 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon_down_mkfs_grace = 2m 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_filestore_osds = false 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub interval = 300 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout:[client] 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout: rgw cache enabled = true 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable ops log = true 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable usage log = true 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.a] 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.b] 2024-04-18T21:50:28.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.c] 2024-04-18T21:50:28.387 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-18T21:50:28.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-18T21:50:28.466 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-18T21:50:28.466 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:28.466 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-18T21:50:28.513 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:28.514 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-18T21:50:28.562 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:50:28.563 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:28.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-18T21:50:28.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-18T21:50:28.627 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:50:28.628 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:28.674 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:50:28.674 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:50:28.674 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-18T21:50:28.675 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-18T21:50:28.697 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:50:28.697 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:28.745 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:50:28.745 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:50:28.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-18T21:50:28.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-18T21:50:28.767 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:50:28.768 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:28.815 INFO:tasks.ceph:Setting up mon nodes... 2024-04-18T21:50:28.815 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-18T21:50:28.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-18T21:50:28.858 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-18T21:50:28.872 INFO:tasks.ceph:Setting up mds nodes... 2024-04-18T21:50:28.872 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-18T21:50:28.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-18T21:50:28.905 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-18T21:50:28.915 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-18T21:50:28.915 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'}} 2024-04-18T21:50:28.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-18T21:50:28.967 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:28.967 INFO:tasks.ceph:role: osd.0 2024-04-18T21:50:28.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:50:28.968 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-18T21:50:29.029 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:29.029 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:29.029 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:29.029 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:29.029 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:29.029 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:29.030 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:29.030 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:29.030 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:29.030 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:50:29.489 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-04-18T21:50:29.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-04-18T21:50:29.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-18T21:50:29.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-18T21:50:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:50:29.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:50:29.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-18T21:50:29.661 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:29.661 INFO:tasks.ceph:role: osd.1 2024-04-18T21:50:29.661 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:50:29.661 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-18T21:50:29.725 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:29.726 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:50:30.192 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-04-18T21:50:30.195 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-04-18T21:50:30.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-18T21:50:30.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-18T21:50:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:50:30.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:50:30.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-18T21:50:30.394 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:30.394 INFO:tasks.ceph:role: osd.2 2024-04-18T21:50:30.394 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:50:30.395 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-18T21:50:30.457 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:30.457 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:30.457 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:30.457 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:30.457 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:30.457 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:30.457 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:30.457 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:30.458 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:30.458 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:50:30.950 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-04-18T21:50:30.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-04-18T21:50:30.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-18T21:50:31.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-18T21:50:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:50:31.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:50:31.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-18T21:50:31.174 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:31.174 INFO:tasks.ceph:role: osd.3 2024-04-18T21:50:31.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:50:31.174 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-18T21:50:31.238 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:31.238 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:31.238 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:31.238 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:31.238 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:31.238 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:31.238 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:31.238 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:31.239 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:31.239 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:50:31.678 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-04-18T21:50:31.681 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-04-18T21:50:31.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-18T21:50:31.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-18T21:50:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:50:31.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:50:31.781 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:31.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:31.912+0000 7f7fcee9f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-18T21:50:31.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:31.916+0000 7f7fcee9f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-18T21:50:31.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:31.916+0000 7f7fcee9f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-18T21:50:31.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:31.916+0000 7f7fcee9f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-18T21:50:31.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:31.916+0000 7f7fcee9f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-18T21:50:31.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:31.916+0000 7f7fcee9f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-18T21:50:34.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-18T21:50:34.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:35.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:35.000+0000 7f6dc9bdb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-18T21:50:35.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:35.000+0000 7f6dc9bdb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-18T21:50:35.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:35.000+0000 7f6dc9bdb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-18T21:50:35.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:35.000+0000 7f6dc9bdb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-18T21:50:35.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:35.000+0000 7f6dc9bdb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-18T21:50:35.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:35.000+0000 7f6dc9bdb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-18T21:50:37.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-18T21:50:38.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:38.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:38.128+0000 7f240a4567c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-18T21:50:38.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:38.128+0000 7f240a4567c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-18T21:50:38.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:38.128+0000 7f240a4567c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-18T21:50:38.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:38.128+0000 7f240a4567c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-18T21:50:38.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:38.128+0000 7f240a4567c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-18T21:50:38.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:38.132+0000 7f240a4567c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-18T21:50:41.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-18T21:50:41.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:41.240+0000 7fa7fdc3b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-18T21:50:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:41.240+0000 7fa7fdc3b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-18T21:50:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:41.240+0000 7fa7fdc3b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-18T21:50:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:41.240+0000 7fa7fdc3b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-18T21:50:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:41.240+0000 7fa7fdc3b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-18T21:50:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-04-18T21:50:41.240+0000 7fa7fdc3b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-18T21:50:44.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-18T21:50:44.322 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-18T21:50:44.337 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:44.337 INFO:tasks.ceph:role: osd.4 2024-04-18T21:50:44.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:50:44.337 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-18T21:50:44.399 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:44.399 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:44.399 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:44.399 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:44.399 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:44.400 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:44.400 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:44.400 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:44.400 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:44.400 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:50:44.874 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-04-18T21:50:44.875 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-04-18T21:50:44.876 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-18T21:50:45.034 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-18T21:50:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:50:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:50:45.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-18T21:50:45.100 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:45.101 INFO:tasks.ceph:role: osd.5 2024-04-18T21:50:45.101 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:50:45.101 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:45.164 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:50:45.627 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-04-18T21:50:45.630 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-04-18T21:50:45.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-18T21:50:45.723 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-18T21:50:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:50:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:50:45.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-18T21:50:45.792 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:45.792 INFO:tasks.ceph:role: osd.6 2024-04-18T21:50:45.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:50:45.792 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-18T21:50:45.859 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:45.859 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:45.859 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:45.860 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:45.860 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:45.860 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:45.860 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:45.860 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:45.860 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:45.860 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:50:46.315 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-04-18T21:50:46.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-04-18T21:50:46.317 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-18T21:50:46.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-18T21:50:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:50:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:50:46.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-18T21:50:46.517 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:46.517 INFO:tasks.ceph:role: osd.7 2024-04-18T21:50:46.517 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:50:46.517 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:46.581 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:50:47.077 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-04-18T21:50:47.079 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-04-18T21:50:47.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-18T21:50:47.227 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-18T21:50:47.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:50:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:50:47.239 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:47.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:47.344+0000 7fa6ea3617c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-18T21:50:47.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:47.348+0000 7fa6ea3617c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-18T21:50:47.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:47.348+0000 7fa6ea3617c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-18T21:50:47.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:47.348+0000 7fa6ea3617c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-18T21:50:47.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:47.348+0000 7fa6ea3617c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-18T21:50:47.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:47.352+0000 7fa6ea3617c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-18T21:50:50.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-18T21:50:50.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:50.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:50.424+0000 7f504d6607c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-18T21:50:50.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:50.424+0000 7f504d6607c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-18T21:50:50.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:50.424+0000 7f504d6607c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-18T21:50:50.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:50.424+0000 7f504d6607c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-18T21:50:50.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:50.424+0000 7f504d6607c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-18T21:50:50.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:50.424+0000 7f504d6607c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-18T21:50:53.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-18T21:50:53.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:53.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:53.528+0000 7f659aa567c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-18T21:50:53.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:53.528+0000 7f659aa567c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-18T21:50:53.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:53.528+0000 7f659aa567c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-18T21:50:53.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:53.528+0000 7f659aa567c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-18T21:50:53.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:53.528+0000 7f659aa567c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-18T21:50:53.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:53.532+0000 7f659aa567c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-18T21:50:56.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-18T21:50:56.563 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:50:56.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:56.644+0000 7f74fb31f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-18T21:50:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:56.644+0000 7f74fb31f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-18T21:50:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:56.644+0000 7f74fb31f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-18T21:50:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:56.644+0000 7f74fb31f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-18T21:50:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:56.644+0000 7f74fb31f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-18T21:50:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-18T21:50:56.644+0000 7f74fb31f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-18T21:50:59.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-18T21:50:59.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-04-18T21:50:59.708 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-18T21:50:59.708 INFO:tasks.ceph:role: osd.8 2024-04-18T21:50:59.708 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:50:59.708 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-18T21:50:59.770 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:50:59.771 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:51:00.223 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-04-18T21:51:00.225 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-04-18T21:51:00.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-8 2024-04-18T21:51:00.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-04-18T21:51:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:51:00.327 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:51:00.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-04-18T21:51:00.385 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-18T21:51:00.385 INFO:tasks.ceph:role: osd.9 2024-04-18T21:51:00.385 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:51:00.385 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-18T21:51:00.447 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:51:00.448 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:51:00.920 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-04-18T21:51:00.922 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-04-18T21:51:00.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-9 2024-04-18T21:51:01.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-04-18T21:51:01.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:51:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:51:01.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-04-18T21:51:01.147 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-18T21:51:01.147 INFO:tasks.ceph:role: osd.10 2024-04-18T21:51:01.147 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:51:01.147 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-18T21:51:01.209 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:51:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:51:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:51:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:51:01.210 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:51:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:51:01.210 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:51:01.210 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:51:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:51:01.210 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:51:01.685 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-04-18T21:51:01.687 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-04-18T21:51:01.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-04-18T21:51:01.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-04-18T21:51:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:51:01.845 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:51:01.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-04-18T21:51:01.898 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-18T21:51:01.898 INFO:tasks.ceph:role: osd.11 2024-04-18T21:51:01.898 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:51:01.898 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T21:51:01.961 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T21:51:01.962 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T21:51:02.386 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-04-18T21:51:02.388 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-04-18T21:51:02.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-11 2024-04-18T21:51:02.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-04-18T21:51:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:51:02.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T21:51:02.528 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:51:02.635 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:02.636+0000 7f1c6d9167c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-04-18T21:51:02.637 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:02.636+0000 7f1c6d9167c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-04-18T21:51:02.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:02.636+0000 7f1c6d9167c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-18T21:51:02.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:02.636+0000 7f1c6d9167c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-18T21:51:02.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:02.636+0000 7f1c6d9167c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-18T21:51:02.638 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:02.640+0000 7f1c6d9167c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-04-18T21:51:05.617 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-04-18T21:51:05.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:51:05.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:05.716+0000 7fbf335da7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-04-18T21:51:05.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:05.716+0000 7fbf335da7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-04-18T21:51:05.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:05.716+0000 7fbf335da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-18T21:51:05.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:05.716+0000 7fbf335da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-18T21:51:05.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:05.716+0000 7fbf335da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-18T21:51:05.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:05.716+0000 7fbf335da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-04-18T21:51:08.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-04-18T21:51:08.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:51:08.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:08.836+0000 7f28916fc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-04-18T21:51:08.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:08.836+0000 7f28916fc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-04-18T21:51:08.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:08.836+0000 7f28916fc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-18T21:51:08.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:08.836+0000 7f28916fc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-18T21:51:08.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:08.836+0000 7f28916fc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-18T21:51:08.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:08.840+0000 7f28916fc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-04-18T21:51:11.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-04-18T21:51:11.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:51:11.958 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:11.956+0000 7f58bddbf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-04-18T21:51:11.958 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:11.960+0000 7f58bddbf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-04-18T21:51:11.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:11.960+0000 7f58bddbf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-18T21:51:11.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:11.960+0000 7f58bddbf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-18T21:51:11.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:11.960+0000 7f58bddbf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-18T21:51:11.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-18T21:51:11.960+0000 7f58bddbf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-04-18T21:51:14.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-04-18T21:51:15.011 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-18T21:51:15.011 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:15.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-18T21:51:15.026 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:15.026 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-18T21:51:15.081 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:15.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-18T21:51:15.137 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:15.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-18T21:51:15.194 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:15.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-18T21:51:15.209 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:15.209 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-18T21:51:15.263 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:15.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-18T21:51:15.315 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:15.315 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-18T21:51:15.367 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:15.368 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-18T21:51:15.419 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:51:15.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-04-18T21:51:15.433 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:51:15.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-04-18T21:51:15.485 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:51:15.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-04-18T21:51:15.537 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:51:15.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-04-18T21:51:15.589 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:15.589 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-18T21:51:15.595 INFO:tasks.ceph:Adding keys to all mons... 2024-04-18T21:51:15.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-18T21:51:15.640 DEBUG:teuthology.orchestra.run.smithi104:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-18T21:51:15.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-18T21:51:15.649 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.0] 2024-04-18T21:51:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCnlSFm/UyFNhAAzVLVF/O/tYUe49rNlY+z6Q== 2024-04-18T21:51:15.649 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.1] 2024-04-18T21:51:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCrlSFm9/VCABAAq/4FCYK8gnikxNjnAujmPg== 2024-04-18T21:51:15.649 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.2] 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCulSFm1/33BxAAukvIAVCk49303HSokFFSgQ== 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.3] 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCxlSFmQMyEDhAAgdU0rPGlDB003TiMfVXjmA== 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.x] 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCklSFm8fU8MxAAZ0Pw33xekcrLxYGxo/3EoQ== 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.4] 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQC3lSFmCrmfFBAAOEZHaNHaF0z1Hy/iAVtHHg== 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.5] 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQC6lSFm20tsGRAAJJPjXw8FZ+KbzXQrIvlHPQ== 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.6] 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQC9lSFmHLi6HxAA9nFAV5eapm643UTm2zE7Cg== 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.7] 2024-04-18T21:51:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDAlSFmqdd8JhAAhizgiGUak0yzDqMhxR8law== 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.8] 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDGlSFmq84SJhAA4pAjeUFVe9JKGLoGULmWjw== 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.9] 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDJlSFmfZ7nKhAAmUIdjuB5+vOOuGrjB/p4gw== 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.10] 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDMlSFmMigbMhAATPsk+wTtZidVqz44UtmJIw== 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.11] 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDPlSFmc/dPORAASa8qmEghhypO5kHJe2ZgWw== 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-04-18T21:51:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCklSFm4Gj+NRAA+z7EYJ5ib96AhCmBfIgXmA== 2024-04-18T21:51:15.652 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.0] 2024-04-18T21:51:15.652 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCnlSFm/UyFNhAAzVLVF/O/tYUe49rNlY+z6Q== 2024-04-18T21:51:15.652 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.1] 2024-04-18T21:51:15.652 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCrlSFm9/VCABAAq/4FCYK8gnikxNjnAujmPg== 2024-04-18T21:51:15.652 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.2] 2024-04-18T21:51:15.652 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCulSFm1/33BxAAukvIAVCk49303HSokFFSgQ== 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.3] 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCxlSFmQMyEDhAAgdU0rPGlDB003TiMfVXjmA== 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.x] 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCklSFm8fU8MxAAZ0Pw33xekcrLxYGxo/3EoQ== 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.4] 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQC3lSFmCrmfFBAAOEZHaNHaF0z1Hy/iAVtHHg== 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.5] 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQC6lSFm20tsGRAAJJPjXw8FZ+KbzXQrIvlHPQ== 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.6] 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQC9lSFmHLi6HxAA9nFAV5eapm643UTm2zE7Cg== 2024-04-18T21:51:15.653 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.7] 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDAlSFmqdd8JhAAhizgiGUak0yzDqMhxR8law== 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.8] 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDGlSFmq84SJhAA4pAjeUFVe9JKGLoGULmWjw== 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.9] 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDJlSFmfZ7nKhAAmUIdjuB5+vOOuGrjB/p4gw== 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.10] 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDMlSFmMigbMhAATPsk+wTtZidVqz44UtmJIw== 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.11] 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDPlSFmc/dPORAASa8qmEghhypO5kHJe2ZgWw== 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-04-18T21:51:15.654 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCklSFm4Gj+NRAA+z7EYJ5ib96AhCmBfIgXmA== 2024-04-18T21:51:15.656 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCnlSFm/UyFNhAAzVLVF/O/tYUe49rNlY+z6Q== 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCrlSFm9/VCABAAq/4FCYK8gnikxNjnAujmPg== 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCulSFm1/33BxAAukvIAVCk49303HSokFFSgQ== 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.3] 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCxlSFmQMyEDhAAgdU0rPGlDB003TiMfVXjmA== 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCklSFm8fU8MxAAZ0Pw33xekcrLxYGxo/3EoQ== 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.4] 2024-04-18T21:51:15.658 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQC3lSFmCrmfFBAAOEZHaNHaF0z1Hy/iAVtHHg== 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.5] 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQC6lSFm20tsGRAAJJPjXw8FZ+KbzXQrIvlHPQ== 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.6] 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQC9lSFmHLi6HxAA9nFAV5eapm643UTm2zE7Cg== 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.7] 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDAlSFmqdd8JhAAhizgiGUak0yzDqMhxR8law== 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.8] 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDGlSFmq84SJhAA4pAjeUFVe9JKGLoGULmWjw== 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.9] 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDJlSFmfZ7nKhAAmUIdjuB5+vOOuGrjB/p4gw== 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.10] 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDMlSFmMigbMhAATPsk+wTtZidVqz44UtmJIw== 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.11] 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDPlSFmc/dPORAASa8qmEghhypO5kHJe2ZgWw== 2024-04-18T21:51:15.659 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-04-18T21:51:15.660 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCklSFm4Gj+NRAA+z7EYJ5ib96AhCmBfIgXmA== 2024-04-18T21:51:15.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.917 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:15.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-18T21:51:16.004 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-18T21:51:16.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-18T21:51:16.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.431 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.603 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.683 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T21:51:16.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-18T21:51:16.767 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-18T21:51:16.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-18T21:51:16.822 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-18T21:51:16.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-18T21:51:16.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-18T21:51:16.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-18T21:51:16.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-18T21:51:16.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-18T21:51:17.078 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-18T21:51:17.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-18T21:51:17.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-18T21:51:17.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-18T21:51:17.209 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:51:17.211 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:51:17.213 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-18T21:51:17.254 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-18T21:51:17.254 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-18T21:51:17.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-18T21:51:17.259 INFO:tasks.ceph.mon.a:Started 2024-04-18T21:51:17.259 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-18T21:51:17.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-18T21:51:17.261 INFO:tasks.ceph.mon.b:Started 2024-04-18T21:51:17.262 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-18T21:51:17.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-18T21:51:17.299 INFO:tasks.ceph.mon.c:Started 2024-04-18T21:51:17.300 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-18T21:51:17.300 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-18T21:51:17.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-18T21:51:17.302 INFO:tasks.ceph.mgr.x:Started 2024-04-18T21:51:17.303 INFO:tasks.ceph:Setting crush tunables to default 2024-04-18T21:51:17.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd crush tunables default 2024-04-18T21:51:17.742 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2024-04-18T21:51:17.762 INFO:tasks.ceph:check_enable_crimson: False 2024-04-18T21:51:17.762 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-18T21:51:17.762 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:17.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-18T21:51:17.775 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:17.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-18T21:51:17.828 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:17.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-18T21:51:17.882 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-18T21:51:17.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-18T21:51:17.894 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:17.892+0000 7f2e7c849180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-18T21:51:17.938 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:17.938 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-18T21:51:17.951 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:17.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-18T21:51:18.004 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:18.004 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-18T21:51:18.060 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-18T21:51:18.060 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-18T21:51:18.116 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:51:18.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-04-18T21:51:18.130 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:51:18.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-04-18T21:51:18.186 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:51:18.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-04-18T21:51:18.242 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-18T21:51:18.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-04-18T21:51:18.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 2a27387f-73d5-4bf2-bb86-80c069a1d6f8 0 2024-04-18T21:51:18.421 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:18.420+0000 7f2e7c849180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-18T21:51:18.623 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:18.620+0000 7f2e7c849180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-18T21:51:18.805 INFO:teuthology.orchestra.run.smithi183.stdout:0 2024-04-18T21:51:18.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 2f6c823f-4b24-4ecf-9561-710723acded2 1 2024-04-18T21:51:19.030 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-04-18T21:51:19.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 05866927-b7ea-44e6-8a5e-f23263011fa4 2 2024-04-18T21:51:19.154 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.152+0000 7f2e7c849180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-18T21:51:19.208 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.204+0000 7f2e7c849180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-18T21:51:19.254 INFO:teuthology.orchestra.run.smithi183.stdout:2 2024-04-18T21:51:19.260 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.256+0000 7f2e7c849180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-18T21:51:19.272 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 1fbdd117-432a-4bef-879e-96823b61820a 3 2024-04-18T21:51:19.336 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.332+0000 7f2e7c849180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-18T21:51:19.387 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.384+0000 7f2e7c849180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-18T21:51:19.462 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.460+0000 7f2e7c849180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-18T21:51:19.478 INFO:teuthology.orchestra.run.smithi183.stdout:3 2024-04-18T21:51:19.496 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new a7f8da99-45f2-4d8f-8c8c-efd9632b0f61 4 2024-04-18T21:51:19.517 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.512+0000 7f2e7c849180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-18T21:51:19.663 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.660+0000 7f2e7c849180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-18T21:51:19.696 INFO:teuthology.orchestra.run.smithi183.stdout:4 2024-04-18T21:51:19.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 2230dfaa-7c85-469b-a95b-594960d77b28 5 2024-04-18T21:51:19.725 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:19.724+0000 7f2e7c849180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-18T21:51:19.921 INFO:teuthology.orchestra.run.smithi183.stdout:5 2024-04-18T21:51:19.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new e7437af9-b161-482e-8939-4d2b4c2da10e 6 2024-04-18T21:51:20.139 INFO:teuthology.orchestra.run.smithi183.stdout:6 2024-04-18T21:51:20.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new a062c3b3-7e00-4bdf-9731-0072826522c9 7 2024-04-18T21:51:20.329 INFO:tasks.ceph.mgr.x.smithi104.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-18T21:51:20.329 INFO:tasks.ceph.mgr.x.smithi104.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-18T21:51:20.329 INFO:tasks.ceph.mgr.x.smithi104.stderr: from numpy import show_config as show_numpy_config 2024-04-18T21:51:20.361 INFO:teuthology.orchestra.run.smithi183.stdout:7 2024-04-18T21:51:20.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new f2473c34-2db3-4806-a6ea-bb46f42c4ab8 8 2024-04-18T21:51:20.386 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:20.384+0000 7f2e7c849180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-18T21:51:20.538 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:20.536+0000 7f2e7c849180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-18T21:51:20.577 INFO:teuthology.orchestra.run.smithi183.stdout:8 2024-04-18T21:51:20.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new f08d755f-cc20-468c-b3ca-cfb88b44bdf1 9 2024-04-18T21:51:20.697 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:20.696+0000 7f2e7c849180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-18T21:51:20.761 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:20.760+0000 7f2e7c849180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-18T21:51:20.802 INFO:teuthology.orchestra.run.smithi183.stdout:9 2024-04-18T21:51:20.819 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 247fad9b-8847-416f-bfbb-9d8b466c3aa6 10 2024-04-18T21:51:20.893 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:20.892+0000 7f2e7c849180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-18T21:51:21.025 INFO:teuthology.orchestra.run.smithi183.stdout:10 2024-04-18T21:51:21.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 4a51c1be-4125-451a-a910-c8e364fe21be 11 2024-04-18T21:51:21.251 INFO:teuthology.orchestra.run.smithi183.stdout:11 2024-04-18T21:51:21.269 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-18T21:51:21.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-18T21:51:21.272 INFO:tasks.ceph.osd.0:Started 2024-04-18T21:51:21.273 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-18T21:51:21.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-18T21:51:21.276 INFO:tasks.ceph.osd.1:Started 2024-04-18T21:51:21.276 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-18T21:51:21.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-18T21:51:21.279 INFO:tasks.ceph.osd.2:Started 2024-04-18T21:51:21.280 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-18T21:51:21.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-18T21:51:21.286 INFO:tasks.ceph.osd.3:Started 2024-04-18T21:51:21.286 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-18T21:51:21.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-18T21:51:21.289 INFO:tasks.ceph.osd.4:Started 2024-04-18T21:51:21.289 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-18T21:51:21.289 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-18T21:51:21.291 INFO:tasks.ceph.osd.5:Started 2024-04-18T21:51:21.291 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-18T21:51:21.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-18T21:51:21.294 INFO:tasks.ceph.osd.6:Started 2024-04-18T21:51:21.294 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-18T21:51:21.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-18T21:51:21.296 INFO:tasks.ceph.osd.7:Started 2024-04-18T21:51:21.296 INFO:tasks.ceph.osd.8:Restarting daemon 2024-04-18T21:51:21.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-04-18T21:51:21.299 INFO:tasks.ceph.osd.8:Started 2024-04-18T21:51:21.299 INFO:tasks.ceph.osd.9:Restarting daemon 2024-04-18T21:51:21.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-04-18T21:51:21.301 INFO:tasks.ceph.osd.9:Started 2024-04-18T21:51:21.301 INFO:tasks.ceph.osd.10:Restarting daemon 2024-04-18T21:51:21.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-04-18T21:51:21.303 INFO:tasks.ceph.osd.10:Started 2024-04-18T21:51:21.304 INFO:tasks.ceph.osd.11:Restarting daemon 2024-04-18T21:51:21.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-04-18T21:51:21.305 INFO:tasks.ceph.osd.11:Started 2024-04-18T21:51:21.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T21:51:21.530 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:21.528+0000 7f2e7c849180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-18T21:51:21.532 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:21.532 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":14,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:51:21.247402+0000","last_up_change":"0.000000","last_in_change":"2024-04-18T21:51:21.247402+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:51:21.550 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-18T21:51:21.550 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-18T21:51:21.604 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:21.600+0000 7f2e7c849180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-18T21:51:21.786 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:21.784+0000 7f2e7c849180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-18T21:51:21.858 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:21.856+0000 7f2e7c849180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-18T21:51:21.894 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:21.892+0000 7f477b5b47c0 -1 Falling back to public interface 2024-04-18T21:51:21.904 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:21.904+0000 7f16495a47c0 -1 Falling back to public interface 2024-04-18T21:51:21.904 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:21.904+0000 7f832995b7c0 -1 Falling back to public interface 2024-04-18T21:51:21.905 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:21.904+0000 7f9d951a07c0 -1 Falling back to public interface 2024-04-18T21:51:21.905 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:21.904+0000 7ff7790567c0 -1 Falling back to public interface 2024-04-18T21:51:21.913 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:21.912+0000 7f712b70d7c0 -1 Falling back to public interface 2024-04-18T21:51:21.914 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:21.912+0000 7f3db395a7c0 -1 Falling back to public interface 2024-04-18T21:51:21.929 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:21.928+0000 7efd305457c0 -1 Falling back to public interface 2024-04-18T21:51:22.024 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:22.019+0000 7f2c009fd7c0 -1 Falling back to public interface 2024-04-18T21:51:22.024 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:22.019+0000 7f86450fa7c0 -1 Falling back to public interface 2024-04-18T21:51:22.024 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:22.019+0000 7f875740b7c0 -1 Falling back to public interface 2024-04-18T21:51:22.025 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:22.019+0000 7fe5a36137c0 -1 Falling back to public interface 2024-04-18T21:51:22.146 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:22.144+0000 7f2e7c849180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-18T21:51:22.208 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:22.208+0000 7f2e7c849180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-18T21:51:22.760 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-18T21:51:22.950 INFO:teuthology.misc.health.smithi097.stdout: 2024-04-18T21:51:22.951 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":14,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:51:21.247402+0000","last_up_change":"0.000000","last_in_change":"2024-04-18T21:51:21.247402+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:51:22.966 DEBUG:teuthology.misc:0 of 12 OSDs are up 2024-04-18T21:51:23.250 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:23.248+0000 7f2e23d91640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-18T21:51:23.251 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-04-18T21:51:23.248+0000 7f2e23d91640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-18T21:51:25.651 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:25.648+0000 7f477b5b47c0 -1 osd.5 0 log_to_monitors true 2024-04-18T21:51:25.689 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:25.688+0000 7f16495a47c0 -1 osd.11 0 log_to_monitors true 2024-04-18T21:51:25.706 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:25.708+0000 7f832995b7c0 -1 osd.8 0 log_to_monitors true 2024-04-18T21:51:25.798 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:25.800+0000 7f9d951a07c0 -1 osd.9 0 log_to_monitors true 2024-04-18T21:51:25.816 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:25.816+0000 7f712b70d7c0 -1 osd.4 0 log_to_monitors true 2024-04-18T21:51:25.821 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:25.820+0000 7ff7790567c0 -1 osd.10 0 log_to_monitors true 2024-04-18T21:51:25.845 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:25.844+0000 7efd305457c0 -1 osd.7 0 log_to_monitors true 2024-04-18T21:51:25.845 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:25.844+0000 7f3db395a7c0 -1 osd.6 0 log_to_monitors true 2024-04-18T21:51:25.873 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:25.871+0000 7f86450fa7c0 -1 osd.3 0 log_to_monitors true 2024-04-18T21:51:25.905 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:25.903+0000 7f875740b7c0 -1 osd.0 0 log_to_monitors true 2024-04-18T21:51:25.933 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:25.927+0000 7f2c009fd7c0 -1 osd.1 0 log_to_monitors true 2024-04-18T21:51:25.938 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:25.935+0000 7fe5a36137c0 -1 osd.2 0 log_to_monitors true 2024-04-18T21:51:26.752 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:26.752+0000 7f83250e1640 -1 osd.8 0 waiting for initial osdmap 2024-04-18T21:51:26.757 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:26.756+0000 7f1644d2a640 -1 osd.11 0 waiting for initial osdmap 2024-04-18T21:51:26.775 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:26.776+0000 7f1641379640 -1 osd.11 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:26.775 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:26.776+0000 7f8321730640 -1 osd.8 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:27.704 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:27.700+0000 7f4778570640 -1 osd.5 0 waiting for initial osdmap 2024-04-18T21:51:27.741 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:27.740+0000 7f476f381640 -1 osd.5 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:27.762 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:27.759+0000 7fe59ed99640 -1 osd.2 0 waiting for initial osdmap 2024-04-18T21:51:27.763 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:27.759+0000 7f8752b91640 -1 osd.0 0 waiting for initial osdmap 2024-04-18T21:51:27.769 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:27.768+0000 7f9d90926640 -1 osd.9 0 waiting for initial osdmap 2024-04-18T21:51:27.769 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:27.768+0000 7efd2bccb640 -1 osd.7 0 waiting for initial osdmap 2024-04-18T21:51:27.790 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:27.792+0000 7f9d8cf75640 -1 osd.9 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:27.790 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:27.788+0000 7efd2831a640 -1 osd.7 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:27.790 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:27.787+0000 7f874f1e0640 -1 osd.0 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:27.803 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:27.799+0000 7fe59b3e8640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:28.832 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:28.832+0000 7ff7747dc640 -1 osd.10 0 waiting for initial osdmap 2024-04-18T21:51:28.963 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:28.964+0000 7ff76ce23640 -1 osd.10 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:29.131 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:29.127+0000 7f3db0916640 -1 osd.6 0 waiting for initial osdmap 2024-04-18T21:51:29.144 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:29.139+0000 7f2bfd9b9640 -1 osd.1 0 waiting for initial osdmap 2024-04-18T21:51:29.177 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:29.175+0000 7f71286c9640 -1 osd.4 0 waiting for initial osdmap 2024-04-18T21:51:29.177 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:29.175+0000 7f8640880640 -1 osd.3 0 waiting for initial osdmap 2024-04-18T21:51:29.258 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:29.255+0000 7f3da7727640 -1 osd.6 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:29.266 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:29.263+0000 7f2bf47ca640 -1 osd.1 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:29.301 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:29.299+0000 7f711f4da640 -1 osd.4 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:29.311 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:29.307+0000 7f8638ec7640 -1 osd.3 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T21:51:30.178 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-18T21:51:30.481 INFO:teuthology.misc.health.smithi097.stdout: 2024-04-18T21:51:30.481 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":19,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:51:29.769135+0000","last_up_change":"2024-04-18T21:51:29.769135+0000","last_in_change":"2024-04-18T21:51:21.247402+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T21:51:28.321879+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6812","nonce":3458443931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6817","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6820","nonce":3458443931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6831","nonce":3458443931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6828","nonce":3458443931}]},"public_addr":"172.21.15.97:6812/3458443931","cluster_addr":"172.21.15.97:6820/3458443931","heartbeat_back_addr":"172.21.15.97:6831/3458443931","heartbeat_front_addr":"172.21.15.97:6828/3458443931","state":["exists","up"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6801","nonce":1179456160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6803","nonce":1179456160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6811","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6815","nonce":1179456160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6807","nonce":1179456160}]},"public_addr":"172.21.15.97:6801/1179456160","cluster_addr":"172.21.15.97:6803/1179456160","heartbeat_back_addr":"172.21.15.97:6815/1179456160","heartbeat_front_addr":"172.21.15.97:6807/1179456160","state":["exists","up"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6813","nonce":598211345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6819","nonce":598211345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6827","nonce":598211345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6821","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6823","nonce":598211345}]},"public_addr":"172.21.15.97:6813/598211345","cluster_addr":"172.21.15.97:6819/598211345","heartbeat_back_addr":"172.21.15.97:6827/598211345","heartbeat_front_addr":"172.21.15.97:6823/598211345","state":["exists","up"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6806","nonce":40393780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6814","nonce":40393780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6829","nonce":40393780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6822","nonce":40393780}]},"public_addr":"172.21.15.97:6806/40393780","cluster_addr":"172.21.15.97:6814/40393780","heartbeat_back_addr":"172.21.15.97:6829/40393780","heartbeat_front_addr":"172.21.15.97:6822/40393780","state":["exists","up"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6809","nonce":3973913623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6811","nonce":3973913623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6815","nonce":3973913623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6813","nonce":3973913623}]},"public_addr":"172.21.15.104:6809/3973913623","cluster_addr":"172.21.15.104:6811/3973913623","heartbeat_back_addr":"172.21.15.104:6815/3973913623","heartbeat_front_addr":"172.21.15.104:6813/3973913623","state":["exists","up"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6801","nonce":1478992259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6803","nonce":1478992259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6807","nonce":1478992259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6805","nonce":1478992259}]},"public_addr":"172.21.15.104:6801/1478992259","cluster_addr":"172.21.15.104:6803/1478992259","heartbeat_back_addr":"172.21.15.104:6807/1478992259","heartbeat_front_addr":"172.21.15.104:6805/1478992259","state":["exists","up"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6817","nonce":462814955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6819","nonce":462814955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6823","nonce":462814955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6821","nonce":462814955}]},"public_addr":"172.21.15.104:6817/462814955","cluster_addr":"172.21.15.104:6819/462814955","heartbeat_back_addr":"172.21.15.104:6823/462814955","heartbeat_front_addr":"172.21.15.104:6821/462814955","state":["exists","up"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6825","nonce":581913590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6827","nonce":581913590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6831","nonce":581913590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6829","nonce":581913590}]},"public_addr":"172.21.15.104:6825/581913590","cluster_addr":"172.21.15.104:6827/581913590","heartbeat_back_addr":"172.21.15.104:6831/581913590","heartbeat_front_addr":"172.21.15.104:6829/581913590","state":["exists","up"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6806","nonce":1020504720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6812","nonce":1020504720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6825","nonce":1020504720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6819","nonce":1020504720}]},"public_addr":"172.21.15.183:6806/1020504720","cluster_addr":"172.21.15.183:6812/1020504720","heartbeat_back_addr":"172.21.15.183:6825/1020504720","heartbeat_front_addr":"172.21.15.183:6819/1020504720","state":["exists","up"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6814","nonce":1194329140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6821","nonce":1194329140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6829","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6830","nonce":1194329140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6827","nonce":1194329140}]},"public_addr":"172.21.15.183:6814/1194329140","cluster_addr":"172.21.15.183:6821/1194329140","heartbeat_back_addr":"172.21.15.183:6830/1194329140","heartbeat_front_addr":"172.21.15.183:6827/1194329140","state":["exists","up"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6813","nonce":1192340806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6817","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6820","nonce":1192340806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6831","nonce":1192340806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6823","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6826","nonce":1192340806}]},"public_addr":"172.21.15.183:6813/1192340806","cluster_addr":"172.21.15.183:6820/1192340806","heartbeat_back_addr":"172.21.15.183:6831/1192340806","heartbeat_front_addr":"172.21.15.183:6826/1192340806","state":["exists","up"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6801","nonce":3144695911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6803","nonce":3144695911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6815","nonce":3144695911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6807","nonce":3144695911}]},"public_addr":"172.21.15.183:6801/3144695911","cluster_addr":"172.21.15.183:6803/3144695911","heartbeat_back_addr":"172.21.15.183:6815/3144695911","heartbeat_front_addr":"172.21.15.183:6807/3144695911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.945475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.895463+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.671915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.809799+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.710699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.754860+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.728380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:51:30.498 DEBUG:teuthology.misc:12 of 12 OSDs are up 2024-04-18T21:51:30.498 INFO:tasks.ceph:Creating RBD pool 2024-04-18T21:51:30.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-18T21:51:31.795 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'rbd' created 2024-04-18T21:51:31.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-18T21:51:32.804 INFO:teuthology.orchestra.run.smithi097.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-18T21:51:32.828 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-18T21:51:32.829 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-18T21:51:32.829 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-18T21:51:32.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-18T21:51:32.829 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-18T21:51:33.160 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:33.175 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":4,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2974935231},{"type":"v1","addr":"172.21.15.104:6833","nonce":2974935231}]},"active_addr":"172.21.15.104:6833/2974935231","active_change":"2024-04-18T21:51:22.231863+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":55437718}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":402111494}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":2835652518}]}]} 2024-04-18T21:51:33.178 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-18T21:51:33.178 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-18T21:51:33.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T21:51:33.451 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:33.451 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:51:32.796052+0000","last_up_change":"2024-04-18T21:51:29.769135+0000","last_in_change":"2024-04-18T21:51:21.247402+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T21:51:28.321879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-18T21:51:30.835497+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.0099999904632568,"score_stable":3.0099999904632568,"optimal_score":0.82999998331069946,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6812","nonce":3458443931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6817","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6820","nonce":3458443931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6831","nonce":3458443931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6828","nonce":3458443931}]},"public_addr":"172.21.15.97:6812/3458443931","cluster_addr":"172.21.15.97:6820/3458443931","heartbeat_back_addr":"172.21.15.97:6831/3458443931","heartbeat_front_addr":"172.21.15.97:6828/3458443931","state":["exists","up"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6801","nonce":1179456160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6803","nonce":1179456160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6811","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6815","nonce":1179456160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6807","nonce":1179456160}]},"public_addr":"172.21.15.97:6801/1179456160","cluster_addr":"172.21.15.97:6803/1179456160","heartbeat_back_addr":"172.21.15.97:6815/1179456160","heartbeat_front_addr":"172.21.15.97:6807/1179456160","state":["exists","up"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6813","nonce":598211345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6819","nonce":598211345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6827","nonce":598211345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6821","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6823","nonce":598211345}]},"public_addr":"172.21.15.97:6813/598211345","cluster_addr":"172.21.15.97:6819/598211345","heartbeat_back_addr":"172.21.15.97:6827/598211345","heartbeat_front_addr":"172.21.15.97:6823/598211345","state":["exists","up"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6806","nonce":40393780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6814","nonce":40393780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6829","nonce":40393780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6822","nonce":40393780}]},"public_addr":"172.21.15.97:6806/40393780","cluster_addr":"172.21.15.97:6814/40393780","heartbeat_back_addr":"172.21.15.97:6829/40393780","heartbeat_front_addr":"172.21.15.97:6822/40393780","state":["exists","up"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6809","nonce":3973913623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6811","nonce":3973913623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6815","nonce":3973913623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6813","nonce":3973913623}]},"public_addr":"172.21.15.104:6809/3973913623","cluster_addr":"172.21.15.104:6811/3973913623","heartbeat_back_addr":"172.21.15.104:6815/3973913623","heartbeat_front_addr":"172.21.15.104:6813/3973913623","state":["exists","up"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6801","nonce":1478992259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6803","nonce":1478992259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6807","nonce":1478992259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6805","nonce":1478992259}]},"public_addr":"172.21.15.104:6801/1478992259","cluster_addr":"172.21.15.104:6803/1478992259","heartbeat_back_addr":"172.21.15.104:6807/1478992259","heartbeat_front_addr":"172.21.15.104:6805/1478992259","state":["exists","up"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6817","nonce":462814955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6819","nonce":462814955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6823","nonce":462814955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6821","nonce":462814955}]},"public_addr":"172.21.15.104:6817/462814955","cluster_addr":"172.21.15.104:6819/462814955","heartbeat_back_addr":"172.21.15.104:6823/462814955","heartbeat_front_addr":"172.21.15.104:6821/462814955","state":["exists","up"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6825","nonce":581913590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6827","nonce":581913590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6831","nonce":581913590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6829","nonce":581913590}]},"public_addr":"172.21.15.104:6825/581913590","cluster_addr":"172.21.15.104:6827/581913590","heartbeat_back_addr":"172.21.15.104:6831/581913590","heartbeat_front_addr":"172.21.15.104:6829/581913590","state":["exists","up"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6806","nonce":1020504720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6812","nonce":1020504720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6825","nonce":1020504720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6819","nonce":1020504720}]},"public_addr":"172.21.15.183:6806/1020504720","cluster_addr":"172.21.15.183:6812/1020504720","heartbeat_back_addr":"172.21.15.183:6825/1020504720","heartbeat_front_addr":"172.21.15.183:6819/1020504720","state":["exists","up"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6814","nonce":1194329140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6821","nonce":1194329140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6829","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6830","nonce":1194329140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6827","nonce":1194329140}]},"public_addr":"172.21.15.183:6814/1194329140","cluster_addr":"172.21.15.183:6821/1194329140","heartbeat_back_addr":"172.21.15.183:6830/1194329140","heartbeat_front_addr":"172.21.15.183:6827/1194329140","state":["exists","up"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6813","nonce":1192340806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6817","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6820","nonce":1192340806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6831","nonce":1192340806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6823","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6826","nonce":1192340806}]},"public_addr":"172.21.15.183:6813/1192340806","cluster_addr":"172.21.15.183:6820/1192340806","heartbeat_back_addr":"172.21.15.183:6831/1192340806","heartbeat_front_addr":"172.21.15.183:6826/1192340806","state":["exists","up"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6801","nonce":3144695911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6803","nonce":3144695911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6815","nonce":3144695911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6807","nonce":3144695911}]},"public_addr":"172.21.15.183:6801/3144695911","cluster_addr":"172.21.15.183:6803/3144695911","heartbeat_back_addr":"172.21.15.183:6815/3144695911","heartbeat_front_addr":"172.21.15.183:6807/3144695911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.945475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.965087+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.895463+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.826302+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.817690+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.671915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.835011+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.809799+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.710699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.754860+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.849759+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.728380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:51:33.468 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-18T21:51:33.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T21:51:33.742 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:33.742 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":22,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:51:32.796052+0000","last_up_change":"2024-04-18T21:51:29.769135+0000","last_in_change":"2024-04-18T21:51:21.247402+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T21:51:28.321879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-18T21:51:30.835497+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.0099999904632568,"score_stable":3.0099999904632568,"optimal_score":0.82999998331069946,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6812","nonce":3458443931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6817","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6820","nonce":3458443931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6831","nonce":3458443931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6828","nonce":3458443931}]},"public_addr":"172.21.15.97:6812/3458443931","cluster_addr":"172.21.15.97:6820/3458443931","heartbeat_back_addr":"172.21.15.97:6831/3458443931","heartbeat_front_addr":"172.21.15.97:6828/3458443931","state":["exists","up"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6801","nonce":1179456160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6803","nonce":1179456160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6811","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6815","nonce":1179456160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6807","nonce":1179456160}]},"public_addr":"172.21.15.97:6801/1179456160","cluster_addr":"172.21.15.97:6803/1179456160","heartbeat_back_addr":"172.21.15.97:6815/1179456160","heartbeat_front_addr":"172.21.15.97:6807/1179456160","state":["exists","up"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6813","nonce":598211345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6819","nonce":598211345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6827","nonce":598211345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6821","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6823","nonce":598211345}]},"public_addr":"172.21.15.97:6813/598211345","cluster_addr":"172.21.15.97:6819/598211345","heartbeat_back_addr":"172.21.15.97:6827/598211345","heartbeat_front_addr":"172.21.15.97:6823/598211345","state":["exists","up"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6806","nonce":40393780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6814","nonce":40393780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6829","nonce":40393780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6822","nonce":40393780}]},"public_addr":"172.21.15.97:6806/40393780","cluster_addr":"172.21.15.97:6814/40393780","heartbeat_back_addr":"172.21.15.97:6829/40393780","heartbeat_front_addr":"172.21.15.97:6822/40393780","state":["exists","up"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6809","nonce":3973913623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6811","nonce":3973913623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6815","nonce":3973913623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6813","nonce":3973913623}]},"public_addr":"172.21.15.104:6809/3973913623","cluster_addr":"172.21.15.104:6811/3973913623","heartbeat_back_addr":"172.21.15.104:6815/3973913623","heartbeat_front_addr":"172.21.15.104:6813/3973913623","state":["exists","up"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6801","nonce":1478992259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6803","nonce":1478992259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6807","nonce":1478992259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6805","nonce":1478992259}]},"public_addr":"172.21.15.104:6801/1478992259","cluster_addr":"172.21.15.104:6803/1478992259","heartbeat_back_addr":"172.21.15.104:6807/1478992259","heartbeat_front_addr":"172.21.15.104:6805/1478992259","state":["exists","up"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6817","nonce":462814955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6819","nonce":462814955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6823","nonce":462814955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6821","nonce":462814955}]},"public_addr":"172.21.15.104:6817/462814955","cluster_addr":"172.21.15.104:6819/462814955","heartbeat_back_addr":"172.21.15.104:6823/462814955","heartbeat_front_addr":"172.21.15.104:6821/462814955","state":["exists","up"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6825","nonce":581913590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6827","nonce":581913590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6831","nonce":581913590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6829","nonce":581913590}]},"public_addr":"172.21.15.104:6825/581913590","cluster_addr":"172.21.15.104:6827/581913590","heartbeat_back_addr":"172.21.15.104:6831/581913590","heartbeat_front_addr":"172.21.15.104:6829/581913590","state":["exists","up"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6806","nonce":1020504720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6812","nonce":1020504720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6825","nonce":1020504720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6819","nonce":1020504720}]},"public_addr":"172.21.15.183:6806/1020504720","cluster_addr":"172.21.15.183:6812/1020504720","heartbeat_back_addr":"172.21.15.183:6825/1020504720","heartbeat_front_addr":"172.21.15.183:6819/1020504720","state":["exists","up"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6814","nonce":1194329140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6821","nonce":1194329140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6829","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6830","nonce":1194329140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6827","nonce":1194329140}]},"public_addr":"172.21.15.183:6814/1194329140","cluster_addr":"172.21.15.183:6821/1194329140","heartbeat_back_addr":"172.21.15.183:6830/1194329140","heartbeat_front_addr":"172.21.15.183:6827/1194329140","state":["exists","up"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6813","nonce":1192340806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6817","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6820","nonce":1192340806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6831","nonce":1192340806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6823","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6826","nonce":1192340806}]},"public_addr":"172.21.15.183:6813/1192340806","cluster_addr":"172.21.15.183:6820/1192340806","heartbeat_back_addr":"172.21.15.183:6831/1192340806","heartbeat_front_addr":"172.21.15.183:6826/1192340806","state":["exists","up"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6801","nonce":3144695911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6803","nonce":3144695911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6815","nonce":3144695911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6807","nonce":3144695911}]},"public_addr":"172.21.15.183:6801/3144695911","cluster_addr":"172.21.15.183:6803/3144695911","heartbeat_back_addr":"172.21.15.183:6815/3144695911","heartbeat_front_addr":"172.21.15.183:6807/3144695911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.945475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.965087+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.895463+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.826302+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.817690+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.671915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.835011+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.809799+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.710699+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.754860+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.849759+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:51:26.728380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:51:33.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-18T21:51:33.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-18T21:51:33.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-18T21:51:33.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-18T21:51:33.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-18T21:51:33.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-18T21:51:33.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-18T21:51:33.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-18T21:51:33.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-18T21:51:33.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-18T21:51:33.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-18T21:51:33.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-18T21:51:34.103 INFO:teuthology.orchestra.run.smithi097.stdout:77309411332 2024-04-18T21:51:34.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-18T21:51:34.107 INFO:teuthology.orchestra.run.smithi097.stdout:81604378628 2024-04-18T21:51:34.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-18T21:51:34.119 INFO:teuthology.orchestra.run.smithi097.stdout:81604378628 2024-04-18T21:51:34.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-18T21:51:34.133 INFO:teuthology.orchestra.run.smithi097.stdout:77309411331 2024-04-18T21:51:34.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-18T21:51:34.138 INFO:teuthology.orchestra.run.smithi097.stdout:77309411330 2024-04-18T21:51:34.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-18T21:51:34.144 INFO:teuthology.orchestra.run.smithi097.stdout:81604378628 2024-04-18T21:51:34.145 INFO:teuthology.orchestra.run.smithi097.stdout:81604378628 2024-04-18T21:51:34.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-18T21:51:34.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-18T21:51:34.158 INFO:teuthology.orchestra.run.smithi097.stdout:81604378629 2024-04-18T21:51:34.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-18T21:51:34.161 INFO:teuthology.orchestra.run.smithi097.stdout:77309411331 2024-04-18T21:51:34.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-18T21:51:34.173 INFO:teuthology.orchestra.run.smithi097.stdout:73014444036 2024-04-18T21:51:34.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-18T21:51:34.190 INFO:teuthology.orchestra.run.smithi097.stdout:73014444036 2024-04-18T21:51:34.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-18T21:51:34.205 INFO:teuthology.orchestra.run.smithi097.stdout:77309411331 2024-04-18T21:51:34.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-18T21:51:34.753 INFO:teuthology.orchestra.run.smithi097.stdout:81604378627 2024-04-18T21:51:34.801 INFO:teuthology.orchestra.run.smithi097.stdout:81604378627 2024-04-18T21:51:34.806 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378628 got 81604378627 for osd.10 2024-04-18T21:51:34.830 INFO:teuthology.orchestra.run.smithi097.stdout:81604378628 2024-04-18T21:51:34.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378628 got 81604378627 for osd.3 2024-04-18T21:51:34.870 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378628 got 81604378628 for osd.6 2024-04-18T21:51:34.870 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:34.871 INFO:teuthology.orchestra.run.smithi097.stdout:81604378629 2024-04-18T21:51:34.877 INFO:teuthology.orchestra.run.smithi097.stdout:77309411332 2024-04-18T21:51:34.886 INFO:teuthology.orchestra.run.smithi097.stdout:77309411330 2024-04-18T21:51:34.905 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378629 got 81604378629 for osd.4 2024-04-18T21:51:34.905 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:34.905 INFO:teuthology.orchestra.run.smithi097.stdout:81604378628 2024-04-18T21:51:34.913 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411330 got 77309411330 for osd.0 2024-04-18T21:51:34.913 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:34.914 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411332 got 77309411332 for osd.2 2024-04-18T21:51:34.915 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:34.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378628 got 81604378628 for osd.1 2024-04-18T21:51:34.931 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:34.939 INFO:teuthology.orchestra.run.smithi097.stdout:77309411331 2024-04-18T21:51:34.952 INFO:teuthology.orchestra.run.smithi097.stdout:77309411331 2024-04-18T21:51:34.965 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.5 2024-04-18T21:51:34.965 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:34.970 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.9 2024-04-18T21:51:34.970 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:34.994 INFO:teuthology.orchestra.run.smithi097.stdout:77309411331 2024-04-18T21:51:35.003 INFO:teuthology.orchestra.run.smithi097.stdout:73014444036 2024-04-18T21:51:35.014 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.7 2024-04-18T21:51:35.014 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:35.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.8 2024-04-18T21:51:35.021 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:35.021 INFO:teuthology.orchestra.run.smithi097.stdout:73014444036 2024-04-18T21:51:35.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.11 2024-04-18T21:51:35.037 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:35.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-18T21:51:35.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-18T21:51:36.107 INFO:teuthology.orchestra.run.smithi097.stdout:81604378628 2024-04-18T21:51:36.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378628 got 81604378628 for osd.10 2024-04-18T21:51:36.124 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:36.147 INFO:teuthology.orchestra.run.smithi097.stdout:81604378628 2024-04-18T21:51:36.164 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378628 got 81604378628 for osd.3 2024-04-18T21:51:36.164 DEBUG:teuthology.parallel:result is None 2024-04-18T21:51:36.164 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-18T21:51:36.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:51:36.481 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:36.481 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:51:36.498 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-18T21:51:36.245611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":327360,"kb_used_data":2812,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132134720,"statfs":{"total":1159641169920,"available":1159305953280,"internally_reserved":0,"allocated":2879488,"data_stored":1249236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":331855165},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625428+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:51:34.625428+0000","last_peered":"2024-04-18T21:51:34.625428+0000","last_clean":"2024-04-18T21:51:34.625428+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:51:34.625428+0000","last_undegraded":"2024-04-18T21:51:34.625428+0000","last_fullsized":"2024-04-18T21:51:34.625428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.629243+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:51:34.629243+0000","last_peered":"2024-04-18T21:51:34.629243+0000","last_clean":"2024-04-18T21:51:34.629243+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:51:34.629243+0000","last_undegraded":"2024-04-18T21:51:34.629243+0000","last_fullsized":"2024-04-18T21:51:34.629243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.579433+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:51:34.579433+0000","last_peered":"2024-04-18T21:51:34.579433+0000","last_clean":"2024-04-18T21:51:34.579433+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:51:34.579433+0000","last_undegraded":"2024-04-18T21:51:34.579433+0000","last_fullsized":"2024-04-18T21:51:34.579433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.025082+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:51:33.025082+0000","last_peered":"2024-04-18T21:51:33.025082+0000","last_clean":"2024-04-18T21:51:33.025082+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:51:33.025082+0000","last_undegraded":"2024-04-18T21:51:33.025082+0000","last_fullsized":"2024-04-18T21:51:33.025082+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:00.751668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625091+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:51:34.625091+0000","last_peered":"2024-04-18T21:51:34.625091+0000","last_clean":"2024-04-18T21:51:34.625091+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:51:34.625091+0000","last_undegraded":"2024-04-18T21:51:34.625091+0000","last_fullsized":"2024-04-18T21:51:34.625091+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.729353+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:51:34.729353+0000","last_peered":"2024-04-18T21:51:34.729353+0000","last_clean":"2024-04-18T21:51:34.729353+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:51:34.729353+0000","last_undegraded":"2024-04-18T21:51:34.729353+0000","last_fullsized":"2024-04-18T21:51:34.729353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.819238+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:51:33.819238+0000","last_peered":"2024-04-18T21:51:33.819238+0000","last_clean":"2024-04-18T21:51:33.819238+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:51:33.819238+0000","last_undegraded":"2024-04-18T21:51:33.819238+0000","last_fullsized":"2024-04-18T21:51:33.819238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625132+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:51:34.625132+0000","last_peered":"2024-04-18T21:51:34.625132+0000","last_clean":"2024-04-18T21:51:34.625132+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:51:34.625132+0000","last_undegraded":"2024-04-18T21:51:34.625132+0000","last_fullsized":"2024-04-18T21:51:34.625132+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'34","reported_seq":67,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.734531+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:51:34.734531+0000","last_peered":"2024-04-18T21:51:34.734531+0000","last_clean":"2024-04-18T21:51:34.734531+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:51:34.734531+0000","last_undegraded":"2024-04-18T21:51:34.734531+0000","last_fullsized":"2024-04-18T21:51:34.734531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":345120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378630,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":364544,"data_stored":223328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":364544,"data_stored":223328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":205328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:51:36.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:51:36.776 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:36.776 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:51:36.793 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-18T21:51:36.245611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":327360,"kb_used_data":2812,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132134720,"statfs":{"total":1159641169920,"available":1159305953280,"internally_reserved":0,"allocated":2879488,"data_stored":1249236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":331855165},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625428+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:51:34.625428+0000","last_peered":"2024-04-18T21:51:34.625428+0000","last_clean":"2024-04-18T21:51:34.625428+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:51:34.625428+0000","last_undegraded":"2024-04-18T21:51:34.625428+0000","last_fullsized":"2024-04-18T21:51:34.625428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.629243+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:51:34.629243+0000","last_peered":"2024-04-18T21:51:34.629243+0000","last_clean":"2024-04-18T21:51:34.629243+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:51:34.629243+0000","last_undegraded":"2024-04-18T21:51:34.629243+0000","last_fullsized":"2024-04-18T21:51:34.629243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.579433+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:51:34.579433+0000","last_peered":"2024-04-18T21:51:34.579433+0000","last_clean":"2024-04-18T21:51:34.579433+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:51:34.579433+0000","last_undegraded":"2024-04-18T21:51:34.579433+0000","last_fullsized":"2024-04-18T21:51:34.579433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.025082+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:51:33.025082+0000","last_peered":"2024-04-18T21:51:33.025082+0000","last_clean":"2024-04-18T21:51:33.025082+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:51:33.025082+0000","last_undegraded":"2024-04-18T21:51:33.025082+0000","last_fullsized":"2024-04-18T21:51:33.025082+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:00.751668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625091+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:51:34.625091+0000","last_peered":"2024-04-18T21:51:34.625091+0000","last_clean":"2024-04-18T21:51:34.625091+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:51:34.625091+0000","last_undegraded":"2024-04-18T21:51:34.625091+0000","last_fullsized":"2024-04-18T21:51:34.625091+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.729353+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:51:34.729353+0000","last_peered":"2024-04-18T21:51:34.729353+0000","last_clean":"2024-04-18T21:51:34.729353+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:51:34.729353+0000","last_undegraded":"2024-04-18T21:51:34.729353+0000","last_fullsized":"2024-04-18T21:51:34.729353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.819238+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:51:33.819238+0000","last_peered":"2024-04-18T21:51:33.819238+0000","last_clean":"2024-04-18T21:51:33.819238+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:51:33.819238+0000","last_undegraded":"2024-04-18T21:51:33.819238+0000","last_fullsized":"2024-04-18T21:51:33.819238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625132+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:51:34.625132+0000","last_peered":"2024-04-18T21:51:34.625132+0000","last_clean":"2024-04-18T21:51:34.625132+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:51:34.625132+0000","last_undegraded":"2024-04-18T21:51:34.625132+0000","last_fullsized":"2024-04-18T21:51:34.625132+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'34","reported_seq":67,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.734531+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:51:34.734531+0000","last_peered":"2024-04-18T21:51:34.734531+0000","last_clean":"2024-04-18T21:51:34.734531+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:51:34.734531+0000","last_undegraded":"2024-04-18T21:51:34.734531+0000","last_fullsized":"2024-04-18T21:51:34.734531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":345120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378630,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":364544,"data_stored":223328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":364544,"data_stored":223328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":205328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:51:36.794 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-18T21:51:36.794 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-18T21:51:36.794 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-18T21:51:36.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-18T21:51:37.098 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:37.098 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-18T21:51:37.116 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-18T21:51:37.116 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-18T21:51:37.150 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-18T21:51:37.150 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd0bf63dd179c05751e4be49477389aa290c73089'}} 2024-04-18T21:51:37.150 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-18T21:51:37.150 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-18T21:51:37.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-18T21:51:37.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:51:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:37.424 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:51:37.441 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-18T21:51:36.245611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":327360,"kb_used_data":2812,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132134720,"statfs":{"total":1159641169920,"available":1159305953280,"internally_reserved":0,"allocated":2879488,"data_stored":1249236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":331855165},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625428+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:51:34.625428+0000","last_peered":"2024-04-18T21:51:34.625428+0000","last_clean":"2024-04-18T21:51:34.625428+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:51:34.625428+0000","last_undegraded":"2024-04-18T21:51:34.625428+0000","last_fullsized":"2024-04-18T21:51:34.625428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.629243+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:51:34.629243+0000","last_peered":"2024-04-18T21:51:34.629243+0000","last_clean":"2024-04-18T21:51:34.629243+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:51:34.629243+0000","last_undegraded":"2024-04-18T21:51:34.629243+0000","last_fullsized":"2024-04-18T21:51:34.629243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.579433+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:51:34.579433+0000","last_peered":"2024-04-18T21:51:34.579433+0000","last_clean":"2024-04-18T21:51:34.579433+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:51:34.579433+0000","last_undegraded":"2024-04-18T21:51:34.579433+0000","last_fullsized":"2024-04-18T21:51:34.579433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.025082+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:51:33.025082+0000","last_peered":"2024-04-18T21:51:33.025082+0000","last_clean":"2024-04-18T21:51:33.025082+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:51:33.025082+0000","last_undegraded":"2024-04-18T21:51:33.025082+0000","last_fullsized":"2024-04-18T21:51:33.025082+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:00.751668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625091+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:51:34.625091+0000","last_peered":"2024-04-18T21:51:34.625091+0000","last_clean":"2024-04-18T21:51:34.625091+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:51:34.625091+0000","last_undegraded":"2024-04-18T21:51:34.625091+0000","last_fullsized":"2024-04-18T21:51:34.625091+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.729353+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:51:34.729353+0000","last_peered":"2024-04-18T21:51:34.729353+0000","last_clean":"2024-04-18T21:51:34.729353+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:51:34.729353+0000","last_undegraded":"2024-04-18T21:51:34.729353+0000","last_fullsized":"2024-04-18T21:51:34.729353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.819238+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:51:33.819238+0000","last_peered":"2024-04-18T21:51:33.819238+0000","last_clean":"2024-04-18T21:51:33.819238+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:51:33.819238+0000","last_undegraded":"2024-04-18T21:51:33.819238+0000","last_fullsized":"2024-04-18T21:51:33.819238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625132+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:51:34.625132+0000","last_peered":"2024-04-18T21:51:34.625132+0000","last_clean":"2024-04-18T21:51:34.625132+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:51:34.625132+0000","last_undegraded":"2024-04-18T21:51:34.625132+0000","last_fullsized":"2024-04-18T21:51:34.625132+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'34","reported_seq":67,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.734531+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:51:34.734531+0000","last_peered":"2024-04-18T21:51:34.734531+0000","last_clean":"2024-04-18T21:51:34.734531+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:51:34.734531+0000","last_undegraded":"2024-04-18T21:51:34.734531+0000","last_fullsized":"2024-04-18T21:51:34.734531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":345120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378630,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":364544,"data_stored":223328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":364544,"data_stored":223328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":205328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:51:37.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:51:37.733 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:37.733 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:51:37.750 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-18T21:51:36.245611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":327360,"kb_used_data":2812,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132134720,"statfs":{"total":1159641169920,"available":1159305953280,"internally_reserved":0,"allocated":2879488,"data_stored":1249236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":331855165},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":6,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000599"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625428+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:51:34.625428+0000","last_peered":"2024-04-18T21:51:34.625428+0000","last_clean":"2024-04-18T21:51:34.625428+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:51:34.625428+0000","last_undegraded":"2024-04-18T21:51:34.625428+0000","last_fullsized":"2024-04-18T21:51:34.625428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.629243+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:51:34.629243+0000","last_peered":"2024-04-18T21:51:34.629243+0000","last_clean":"2024-04-18T21:51:34.629243+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:51:34.629243+0000","last_undegraded":"2024-04-18T21:51:34.629243+0000","last_fullsized":"2024-04-18T21:51:34.629243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.579433+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:51:34.579433+0000","last_peered":"2024-04-18T21:51:34.579433+0000","last_clean":"2024-04-18T21:51:34.579433+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:51:34.579433+0000","last_undegraded":"2024-04-18T21:51:34.579433+0000","last_fullsized":"2024-04-18T21:51:34.579433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.025082+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:51:33.025082+0000","last_peered":"2024-04-18T21:51:33.025082+0000","last_clean":"2024-04-18T21:51:33.025082+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:51:33.025082+0000","last_undegraded":"2024-04-18T21:51:33.025082+0000","last_fullsized":"2024-04-18T21:51:33.025082+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:00.751668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625091+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:51:34.625091+0000","last_peered":"2024-04-18T21:51:34.625091+0000","last_clean":"2024-04-18T21:51:34.625091+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:51:34.625091+0000","last_undegraded":"2024-04-18T21:51:34.625091+0000","last_fullsized":"2024-04-18T21:51:34.625091+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.729353+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:51:34.729353+0000","last_peered":"2024-04-18T21:51:34.729353+0000","last_clean":"2024-04-18T21:51:34.729353+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:51:34.729353+0000","last_undegraded":"2024-04-18T21:51:34.729353+0000","last_fullsized":"2024-04-18T21:51:34.729353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.819238+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:51:33.819238+0000","last_peered":"2024-04-18T21:51:33.819238+0000","last_clean":"2024-04-18T21:51:33.819238+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:51:33.819238+0000","last_undegraded":"2024-04-18T21:51:33.819238+0000","last_fullsized":"2024-04-18T21:51:33.819238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625132+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:51:34.625132+0000","last_peered":"2024-04-18T21:51:34.625132+0000","last_clean":"2024-04-18T21:51:34.625132+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:51:34.625132+0000","last_undegraded":"2024-04-18T21:51:34.625132+0000","last_fullsized":"2024-04-18T21:51:34.625132+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'34","reported_seq":67,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.734531+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:51:34.734531+0000","last_peered":"2024-04-18T21:51:34.734531+0000","last_clean":"2024-04-18T21:51:34.734531+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:51:34.734531+0000","last_undegraded":"2024-04-18T21:51:34.734531+0000","last_fullsized":"2024-04-18T21:51:34.734531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":345120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378630,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":364544,"data_stored":223328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":364544,"data_stored":223328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":76464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":205328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:51:37.751 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-18T21:51:37.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 23 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:fsid 9a7b119f-53a7-4e46-8d6e-229eac84ebee 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-04-18T21:51:17.630312+0000 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:modified 2024-04-18T21:51:33.804996+0000 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:crush_version 5 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:full_ratio 0.95 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:backfillfull_ratio 0.9 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:nearfull_ratio 0.85 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:require_min_compat_client luminous 2024-04-18T21:51:38.034 INFO:teuthology.orchestra.run.smithi097.stdout:min_compat_client jewel 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:require_osd_release squid 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:stretch_mode_enabled false 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 application rbd read_balance_score 3.01 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:max_osd 12 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6808/3458443931,v1:172.21.15.97:6812/3458443931] [v2:172.21.15.97:6817/3458443931,v1:172.21.15.97:6820/3458443931] exists,up 2a27387f-73d5-4bf2-bb86-80c069a1d6f8 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6800/1179456160,v1:172.21.15.97:6801/1179456160] [v2:172.21.15.97:6802/1179456160,v1:172.21.15.97:6803/1179456160] exists,up 2f6c823f-4b24-4ecf-9561-710723acded2 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6809/598211345,v1:172.21.15.97:6813/598211345] [v2:172.21.15.97:6816/598211345,v1:172.21.15.97:6819/598211345] exists,up 05866927-b7ea-44e6-8a5e-f23263011fa4 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 up in weight 1 up_from 19 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6804/40393780,v1:172.21.15.97:6806/40393780] [v2:172.21.15.97:6810/40393780,v1:172.21.15.97:6814/40393780] exists,up 1fbdd117-432a-4bef-879e-96823b61820a 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6808/3973913623,v1:172.21.15.104:6809/3973913623] [v2:172.21.15.104:6810/3973913623,v1:172.21.15.104:6811/3973913623] exists,up a7f8da99-45f2-4d8f-8c8c-efd9632b0f61 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6800/1478992259,v1:172.21.15.104:6801/1478992259] [v2:172.21.15.104:6802/1478992259,v1:172.21.15.104:6803/1478992259] exists,up 2230dfaa-7c85-469b-a95b-594960d77b28 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 up in weight 1 up_from 19 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6816/462814955,v1:172.21.15.104:6817/462814955] [v2:172.21.15.104:6818/462814955,v1:172.21.15.104:6819/462814955] exists,up e7437af9-b161-482e-8939-4d2b4c2da10e 2024-04-18T21:51:38.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6824/581913590,v1:172.21.15.104:6825/581913590] [v2:172.21.15.104:6826/581913590,v1:172.21.15.104:6827/581913590] exists,up a062c3b3-7e00-4bdf-9731-0072826522c9 2024-04-18T21:51:38.036 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6804/1020504720,v1:172.21.15.183:6806/1020504720] [v2:172.21.15.183:6808/1020504720,v1:172.21.15.183:6812/1020504720] exists,up f2473c34-2db3-4806-a6ea-bb46f42c4ab8 2024-04-18T21:51:38.036 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6810/1194329140,v1:172.21.15.183:6814/1194329140] [v2:172.21.15.183:6818/1194329140,v1:172.21.15.183:6821/1194329140] exists,up f08d755f-cc20-468c-b3ca-cfb88b44bdf1 2024-04-18T21:51:38.036 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6809/1192340806,v1:172.21.15.183:6813/1192340806] [v2:172.21.15.183:6817/1192340806,v1:172.21.15.183:6820/1192340806] exists,up 247fad9b-8847-416f-bfbb-9d8b466c3aa6 2024-04-18T21:51:38.036 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6800/3144695911,v1:172.21.15.183:6801/3144695911] [v2:172.21.15.183:6802/3144695911,v1:172.21.15.183:6803/3144695911] exists,up 4a51c1be-4125-451a-a910-c8e364fe21be 2024-04-18T21:51:38.051 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6808/3458443931,v1:172.21.15.97:6812/3458443931] [v2:172.21.15.97:6817/3458443931,v1:172.21.15.97:6820/3458443931] exists,up 2a27387f-73d5-4bf2-bb86-80c069a1d6f8', 'osd.1 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6800/1179456160,v1:172.21.15.97:6801/1179456160] [v2:172.21.15.97:6802/1179456160,v1:172.21.15.97:6803/1179456160] exists,up 2f6c823f-4b24-4ecf-9561-710723acded2', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6809/598211345,v1:172.21.15.97:6813/598211345] [v2:172.21.15.97:6816/598211345,v1:172.21.15.97:6819/598211345] exists,up 05866927-b7ea-44e6-8a5e-f23263011fa4', 'osd.3 up in weight 1 up_from 19 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6804/40393780,v1:172.21.15.97:6806/40393780] [v2:172.21.15.97:6810/40393780,v1:172.21.15.97:6814/40393780] exists,up 1fbdd117-432a-4bef-879e-96823b61820a', 'osd.4 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6808/3973913623,v1:172.21.15.104:6809/3973913623] [v2:172.21.15.104:6810/3973913623,v1:172.21.15.104:6811/3973913623] exists,up a7f8da99-45f2-4d8f-8c8c-efd9632b0f61', 'osd.5 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6800/1478992259,v1:172.21.15.104:6801/1478992259] [v2:172.21.15.104:6802/1478992259,v1:172.21.15.104:6803/1478992259] exists,up 2230dfaa-7c85-469b-a95b-594960d77b28', 'osd.6 up in weight 1 up_from 19 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6816/462814955,v1:172.21.15.104:6817/462814955] [v2:172.21.15.104:6818/462814955,v1:172.21.15.104:6819/462814955] exists,up e7437af9-b161-482e-8939-4d2b4c2da10e', 'osd.7 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6824/581913590,v1:172.21.15.104:6825/581913590] [v2:172.21.15.104:6826/581913590,v1:172.21.15.104:6827/581913590] exists,up a062c3b3-7e00-4bdf-9731-0072826522c9', 'osd.8 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6804/1020504720,v1:172.21.15.183:6806/1020504720] [v2:172.21.15.183:6808/1020504720,v1:172.21.15.183:6812/1020504720] exists,up f2473c34-2db3-4806-a6ea-bb46f42c4ab8', 'osd.9 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6810/1194329140,v1:172.21.15.183:6814/1194329140] [v2:172.21.15.183:6818/1194329140,v1:172.21.15.183:6821/1194329140] exists,up f08d755f-cc20-468c-b3ca-cfb88b44bdf1', 'osd.10 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6809/1192340806,v1:172.21.15.183:6813/1192340806] [v2:172.21.15.183:6817/1192340806,v1:172.21.15.183:6820/1192340806] exists,up 247fad9b-8847-416f-bfbb-9d8b466c3aa6', 'osd.11 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6800/3144695911,v1:172.21.15.183:6801/3144695911] [v2:172.21.15.183:6802/3144695911,v1:172.21.15.183:6803/3144695911] exists,up 4a51c1be-4125-451a-a910-c8e364fe21be'] 2024-04-18T21:51:38.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-18T21:51:38.156 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:38.156 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mon.a", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster": "ceph", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "admin_socket_mode": "", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_client_required": "cephx, none", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_cluster_required": "cephx", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_debug": "true", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_service_required": "cephx", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "auth_supported": "cephx", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_aio": "true", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-18T21:51:38.157 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_aio_poll_ms": "250", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_aio_reap_max": "16", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_async_discard": "false", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_block_size": "4096", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_debug_aio": "false", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_debug_inflight_ios": "false", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_enable_discard": "false", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_flock_retry": "3", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_inject_crash": "0", 2024-04-18T21:51:38.158 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_ioring": "false", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_ioring_hipri": "false", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bdev_type": "", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_alloc_size": "1048576", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_allocator": "hybrid", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_buffered_io": "true", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_check_for_zeros": "false", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-18T21:51:38.159 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_compact_log_sync": "false", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_max_log_runway": "4194304", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_max_prefetch": "1048576", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_min_flush_size": "524288", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_min_log_runway": "1048576", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_replay_recovery": "false", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_sync_write": "false", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-18T21:51:38.160 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_allocation_from_file": "true", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_allocator": "hybrid", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_create": "true", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_db_create": "false", 2024-04-18T21:51:38.161 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_db_path": "", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_db_size": "0", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_path": "", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_preallocate_file": "false", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_size": "107374182400", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_wal_create": "false", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_wal_path": "", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_block_wal_size": "100663296", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bluefs": "true", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-18T21:51:38.162 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_autotune": "true", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_size": "0", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-18T21:51:38.163 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_type": "2q", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_clone_cow": "true", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_mode": "none", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_csum_type": "crc32c", 2024-04-18T21:51:38.164 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_freelist": "false", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-18T21:51:38.165 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_small_allocations": "0", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_default_buffered_read": "true", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_default_buffered_write": "false", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-18T21:51:38.166 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fail_eio": "false", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_mount": "false", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-18T21:51:38.167 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_umount": "false", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_ignore_data_csum": "false", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_log_op_age": "5.000000", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_alloc_size": "0", 2024-04-18T21:51:38.168 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_blob_size": "0", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_max_deferred_txc": "32", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_min_alloc_size": "0", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_nid_prealloc": "1024", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_retry_disk_reads": "3", 2024-04-18T21:51:38.169 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_rocksdb_cf": "true", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_spdk_mem": "512", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-18T21:51:38.170 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_tracing": "false", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_zero_block_detection": "false", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-18T21:51:38.171 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_cluster_require_signatures": "false", 2024-04-18T21:51:38.172 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_cluster_require_version": "2", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_require_signatures": "false", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_require_version": "2", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_service_require_signatures": "false", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_service_require_version": "2", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "cephx_sign_messages": "true", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "chdir": "", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "client_acl_type": "", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "client_asio_thread_count": "2", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "client_cache_mid": "0.750000", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "client_cache_size": "16384", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "client_caps_release_delay": "5", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "client_check_pool_perm": "true", 2024-04-18T21:51:38.173 INFO:teuthology.orchestra.run.smithi097.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_debug_force_sync_read": "false", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_debug_getattr_caps": "false", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_debug_inject_tick_delay": "0", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_die_on_failed_remount": "false", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_dirsize_rbytes": "true", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_force_lazyio": "false", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_fs": "", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_inject_release_failure": "false", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_max_inline_size": "4096", 2024-04-18T21:51:38.174 INFO:teuthology.orchestra.run.smithi097.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mds_namespace": "", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_metadata": "", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mount_gid": "-1", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mount_timeout": "300", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mount_uid": "-1", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_mountpoint": "/", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_notify_timeout": "10", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc": "true", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_max_dirty": "104857600", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_max_objects": "1000", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_size": "209715200", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_oc_target_dirty": "8388608", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_permissions": "true", 2024-04-18T21:51:38.175 INFO:teuthology.orchestra.run.smithi097.stdout: "client_quota": "true", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_quota_df": "true", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_readahead_max_bytes": "0", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_readahead_max_periods": "4", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_readahead_min": "131072", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_reconnect_stale": "false", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_shutdown_timeout": "30", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_snapdir": ".snap", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_tick_interval": "1", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_trace": "", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_try_dentry_invalidate": "false", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_use_faked_inos": "false", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "client_use_random_mds": "false", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_graylog": "false", 2024-04-18T21:51:38.176 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_graylog_port": "12201", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_monitors": "default=true", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_syslog": "false", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "clog_to_syslog_level": "info", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "-", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_network": "", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_network_interface": "", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "compressor_zlib_isal": "false", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "compressor_zlib_level": "5", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "compressor_zlib_winsize": "-15", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "compressor_zstd_level": "1", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-18T21:51:38.177 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_alien_op_num_threads": "6", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "crimson_seastar_cpu_cores": "", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_location": "", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_location_hook": "", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_location_hook_timeout": "10", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "daemonize": "false", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "daos_pool": "tank", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_alienstore": "0/5", 2024-04-18T21:51:38.178 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_allow_any_pool_priority": "false", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_asok": "1/5", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_asok_assert_abort": "false", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_asserts_on_shutdown": "true", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_auth": "20/20", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_bdev": "1/3", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_bluefs": "1/5", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_bluestore": "1/5", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_buffer": "0/1", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_ceph_exporter": "1/5", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_cephfs_mirror": "0/5", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_cephsqlite": "0/5", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_client": "0/5", 2024-04-18T21:51:38.179 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_compressor": "1/5", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_context": "0/1", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_crimson_interrupt": "0/5", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_crush": "1/1", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_crypto": "1/5", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_cyanstore": "0/5", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_deliberately_leak_memory": "false", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_disable_randomized_ping": "false", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_dpdk": "1/5", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_eventtrace": "1/5", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_filer": "0/1", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_filestore": "1/3", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_finisher": "1/1", 2024-04-18T21:51:38.180 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_fuse": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_heartbeat_testing_span": "0", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_heartbeatmap": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_javaclient": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_journal": "1/3", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_journaler": "0/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_kstore": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_lockdep": "0/1", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mclock": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_balancer": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_locker": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_log": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_log_expire": "1/5", 2024-04-18T21:51:38.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_migrator": "1/5", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mds_quiesce": "3/5", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_memstore": "1/5", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mgr": "2/5", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mgrc": "1/5", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_mon": "20/20", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_monc": "0/10", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_ms": "1/1", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_none": "0/5", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_objclass": "0/5", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_objectcacher": "0/5", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_objecter": "0/1", 2024-04-18T21:51:38.182 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_optracker": "0/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_osd": "1/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_paxos": "20/20", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_perfcounter": "1/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_prioritycache": "1/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rados": "0/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rbd": "0/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rbd_mirror": "0/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rbd_pwl": "0/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rbd_replay": "0/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_refs": "0/0", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_reserver": "1/1", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw": "1/5", 2024-04-18T21:51:38.183 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_access": "1/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_datacache": "1/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_dbstore": "1/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_flight": "1/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rgw_sync": "1/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_rocksdb": "4/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore": "0/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_backref": "0/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_cache": "0/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_cleaner": "0/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_device": "0/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_epm": "0/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-18T21:51:38.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_journal": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_lba": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_odata": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_omap": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_onode": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_t": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_seastore_tm": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_striper": "0/1", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_test": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_throttle": "1/1", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_timer": "0/1", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_tp": "0/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "debug_trace": "1/5", 2024-04-18T21:51:38.185 INFO:teuthology.orchestra.run.smithi097.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "device_failure_prediction_mode": "none", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "err_to_graylog": "false", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "err_to_journald": "false", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "err_to_stderr": "true", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "err_to_syslog": "false", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event_tracing": "false", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_addr": "0.0.0.0", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_http_port": "9926", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_prio_limit": "5", 2024-04-18T21:51:38.186 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_sort_metrics": "true", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "exporter_stats_period": "5", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "fake_statfs_for_testing": "0", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "fatal_signal_handlers": "true", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filer_max_purge_ops": "10", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filer_max_truncate_ops": "128", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_apply_finisher_threads": "1", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_blackhole": "false", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_btrfs_clone_range": "true", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_btrfs_snap": "true", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_caller_concurrency": "10", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_collect_device_partition_information": "true", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_commit_timeout": "600.000000", 2024-04-18T21:51:38.187 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_debug_inject_read_err": "false", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_debug_omap_check": "false", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_debug_verify_split": "false", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_dump_file": "", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fadvise": "true", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fail_eio": "true", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fd_cache_shards": "16", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fd_cache_size": "128", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fiemap": "false", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fiemap_threshold": "4096", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-18T21:51:38.188 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_inject_stall": "0", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_journal_parallel": "false", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_journal_trailing": "false", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_journal_writeahead": "false", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_kill_at": "0", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattrs": "0", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-18T21:51:38.189 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_xattr_value_size": "0", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_merge_threshold": "-10", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_odsync_write": "false", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_omap_backend": "rocksdb", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_omap_backend_path": "", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-18T21:51:38.190 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_op_thread_timeout": "60", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_op_threads": "2", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_punch_hole": "false", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_queue_max_ops": "50", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_seek_data_hole": "false", 2024-04-18T21:51:38.191 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_sloppy_crc": "false", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_splice": "false", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_split_multiple": "2", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_split_rand_factor": "20", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_update_to": "1000", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_enable": "true", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-18T21:51:38.192 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_xfs_extsize": "false", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "filestore_zfs_snap": "false", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "fio_dir": "/tmp/fio", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "fsid": "9a7b119f-53a7-4e46-8d6e-229eac84ebee", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_allow_other": "true", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_atomic_o_trunc": "true", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_big_writes": "true", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_debug": "false", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_default_permissions": "false", 2024-04-18T21:51:38.193 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_disable_pagecache": "false", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_max_write": "0", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_multithreaded": "true", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_require_active_mds": "true", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_set_user_groups": "true", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_splice_move": "true", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_splice_read": "true", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_splice_write": "true", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "fuse_use_invalidate_cb": "true", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "gss_target_name": "ceph", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_file": "", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_inject_failure": "0", 2024-04-18T21:51:38.194 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_interval": "5", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "host": "smithi097", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-18T21:51:38.195 INFO:teuthology.orchestra.run.smithi097.stdout: "inject_early_sigterm": "false", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "jaeger_agent_port": "6799", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "jaeger_tracing_enable": "false", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_aio": "true", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_align_min_size": "65536", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_block_align": "true", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_block_size": "4096", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_dio": "true", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_discard": "false", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_force_aio": "false", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_ignore_corruption": "false", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_max_write_bytes": "10485760", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_max_write_entries": "100", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_replay_from": "0", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-18T21:51:38.196 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_write_header_frequency": "0", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "journal_zero_on_create": "false", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "journaler_prefetch_periods": "10", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "journaler_prezero_periods": "5", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "journaler_write_head_interval": "15", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "keyfile": "", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_backend": "rocksdb", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_default_stripe_size": "65536", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_fsck_on_mount": "false", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-18T21:51:38.197 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_max_bytes": "67108864", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_max_ops": "512", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_nid_prealloc": "1024", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_onode_map_size": "1024", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_sync_submit_transaction": "false", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "kstore_sync_transaction": "false", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "librados_thread_count": "2", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "lockdep": "false", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "lockdep_force_backtrace": "false", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "log_coarse_timestamps": "true", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "log_flush_on_exit": "false", 2024-04-18T21:51:38.198 INFO:teuthology.orchestra.run.smithi097.stdout: "log_graylog_host": "127.0.0.1", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_graylog_port": "12201", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_max_new": "1000", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_max_recent": "10000", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_stderr_prefix": "", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_stop_at_utilization": "0.970000", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": "true", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_graylog": "false", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_journald": "false", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_stderr": "false", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_syslog": "false", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "max_rotating_auth_attempts": "10", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_action_on_write_error": "1", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_alternate_name_max": "8192", 2024-04-18T21:51:38.199 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_asio_thread_count": "2", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_export_pin": "true", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_fragment_dirs": "true", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_fragment_interval": "5", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_interval": "10", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_max": "-1", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_max_until": "-1", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_merge_size": "50", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_midchunk": "0.300000", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-18T21:51:38.200 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_min_start": "0.200000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_minchunk": "0.001000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_mode": "0", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_need_max": "1.200000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_need_min": "0.800000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_overload_epochs": "2", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_split_bits": "3", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_split_size": "10000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_target_decay": "10.000000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-18T21:51:38.201 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_beacon_grace": "15.000000", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_beacon_interval": "4.000000", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_mid": "0.700000", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_delay": "0", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_release_free_interval": "10", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_reservation": "0.050000", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_trim_interval": "1", 2024-04-18T21:51:38.202 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache_trim_threshold": "262144", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_client_prealloc_inos": "1000", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_connect_bootstrapping": "false", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_damage_table_max_entries": "10000", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_debug_auth_pins": "false", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_debug_frag": "false", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_debug_scatterstat": "false", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_debug_subtrees": "false", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_decay_halflife": "5.000000", 2024-04-18T21:51:38.203 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_default_dir_hash": "2", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_defer_session_stale": "true", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_deny_all_reconnect": "false", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dir_keys_per_op": "16384", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dir_max_commit_size": "10", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dir_max_entries": "0", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dir_prefetch": "true", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dump_cache_on_map": "false", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_early_reply": "true", 2024-04-18T21:51:38.204 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_enable_op_tracker": "true", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_enforce_unique_name": "true", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_export_ephemeral_random": "true", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_health_summarize_threshold": "10", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-18T21:51:38.205 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_health_dummy": "false", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_migrator_session_race": "false", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_join_fs": "", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_journal_format": "1", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_create_at": "0", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_export_at": "0", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_import_at": "0", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_journal_at": "0", 2024-04-18T21:51:38.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_journal_expire_at": "0", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_journal_replay_at": "0", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_link_at": "0", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_mdstable_at": "0", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_openc_at": "0", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_rename_at": "0", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_shutdown_at": "0", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_event_large_threshold": "524288", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_events_per_segment": "1024", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_max_events": "-1", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_max_segments": "128", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_pause": "false", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_segment_size": "0", 2024-04-18T21:51:38.207 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_trim_threshold": "128", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log_warn_factor": "2.000000", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_caps_per_client": "1048576", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_completed_flushes": "100000", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_completed_requests": "100000", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_export_size": "20971520", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_file_recover": "32", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_purge_files": "64", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_purge_ops": "8192", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-18T21:51:38.208 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_max_snaps_per_dir": "100", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_metrics_update_interval": "2", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_min_caps_per_client": "100", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_min_caps_working_set": "10000", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_numa_node": "-1", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_complaint_time": "30.000000", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_history_duration": "600", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_history_size": "20", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_history_slow_op_size": "20", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_op_log_threshold": "5", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_ping_grace": "15", 2024-04-18T21:51:38.209 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_ping_interval": "5", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_max_caps": "30000", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_recall_warning_threshold": "262144", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_replay_interval": "1.000000", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_root_ino_gid": "0", 2024-04-18T21:51:38.210 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_root_ino_uid": "0", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_shutdown_check": "0", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_skip_ino": "0", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-18T21:51:38.211 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_snap_max_uid": "4294967294", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_snap_min_uid": "0", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_snap_rstat": "false", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_standby_replay_damaged": "false", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_symlink_recovery": "true", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_thrash_exports": "0", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_thrash_fragments": "0", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_tick_interval": "5.000000", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_valgrind_exit": "false", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_verify_backtrace": "1", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_verify_scatter": "false", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_wipe_sessions": "false", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool_debug": "false", 2024-04-18T21:51:38.212 INFO:teuthology.orchestra.run.smithi097.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "memstore_device_bytes": "1073741824", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "memstore_page_set": "false", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "memstore_page_size": "65536", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_client_bytes": "134217728", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_client_messages": "512", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_disabled_modules": "", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_max_pg_num_change": "128", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_mds_bytes": "134217728", 2024-04-18T21:51:38.213 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_mds_messages": "128", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_mon_bytes": "134217728", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_mon_messages": "128", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_osd_bytes": "536870912", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_osd_messages": "8192", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_pool": "true", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_standby_modules": "true", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_stats_period": "5", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_stats_threshold": "5", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_tick_period": "2", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-18T21:51:38.214 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_allow_pool_delete": "true", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_allow_pool_size_one": "true", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_auth_validate_all_caps": "true", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_bytes": "104857600", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_directed_command_retry": "2", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_hunt_parallel": "3", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_log_interval": "1.000000", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-18T21:51:38.215 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_ping_interval": "10.000000", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_client_target_rank": "-1", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_file_level": "debug", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_file": "true", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_journald": "false", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-18T21:51:38.216 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_compact_on_bootstrap": "false", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_compact_on_start": "false", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_compact_on_trim": "true", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_cpu_threads": "4", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_crush_min_required_version": "hammer", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_daemon_bytes": "419430400", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_data_avail_crit": "5", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_data_avail_warn": "5", 2024-04-18T21:51:38.217 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_data_size_warn": "16106127360", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_dump_json": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_dump_transactions": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_extra_checks": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_no_require_reef": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_no_require_squid": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-18T21:51:38.218 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_down_mkfs_grace": "120", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_down_uptime_grace": "60", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_election_default_strategy": "1", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_election_timeout": "5.000000", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_elector_ping_divisor": "2", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_enable_op_tracker": "true", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_fake_pool_delete": "false", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_force_quorum_join": "false", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_globalid_prealloc": "10000", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_detail_to_clog": "false", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_log_update_period": "5", 2024-04-18T21:51:38.219 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_max_detail": "50", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_to_clog": "true", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_to_clog_interval": "600", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_host": "172.21.15.97,172.21.15.104,172.21.15.183", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_host_override": "", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_initial_members": "", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_lease": "5.000000", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-18T21:51:38.220 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_log_full_interval": "50", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_log_max": "10000", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_log_max_summary": "50", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_log_epochs": "500", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_osd": "10000", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_pg_per_osd": "10000", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_pool_pg_num": "65536", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mds_force_trim_to": "0", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mds_skip_sanity": "false", 2024-04-18T21:51:38.221 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_memory_autotune": "true", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_memory_target": "2147483648", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_beacon_grace": "30", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_digest_period": "5", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_inactive_grace": "60", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_min_osdmap_epochs": "500", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_complaint_time": "30", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_history_duration": "600", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_history_size": "20", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_history_slow_op_size": "20", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_op_log_threshold": "5", 2024-04-18T21:51:38.222 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_auto_mark_in": "false", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_cache_size": "500", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_down_out_interval": "600", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-18T21:51:38.223 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_force_trim_to": "0", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_min_down_reporters": "2", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-18T21:51:38.224 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_report_timeout": "900", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_stuck_threshold": "60", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-18T21:51:38.225 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_probe_timeout": "2.000000", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_reweight_max_change": "0.050000", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_reweight_max_osds": "4", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_interval": "300", 2024-04-18T21:51:38.226 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_max_keys": "100", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_scrub_timeout": "300", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_session_timeout": "300", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_smart_report_timeout": "5", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stat_smooth_intervals": "6", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stretch_pool_min_size": "2", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stretch_pool_size": "4", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_debug": "false", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_provider_kill_at": "0", 2024-04-18T21:51:38.227 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_requester_kill_at": "0", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_sync_timeout": "60.000000", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_target_pg_per_osd": "100", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_tick_interval": "5", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_timecheck_interval": "300.000000", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_use_min_delay_socket": "false", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_older_version_delay": "604800", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-18T21:51:38.228 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_misplaced": "false", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_older_version": "true", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "monmap": "", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-18T21:51:38.229 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "motr_tracing_enabled": "false", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_op_threads": "3", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_cm": "false", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_device_name": "", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_dscp": "96", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_local_gid": "", 2024-04-18T21:51:38.230 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_port_num": "1", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_sl": "3", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_support_srq": "true", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_rdma_type": "ib", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_async_reap_threshold": "5", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_before_connect": "false", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_ipv4": "true", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_ipv6": "false", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_msgr1": "true", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_msgr2": "true", 2024-04-18T21:51:38.231 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_port_max": "7568", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_port_min": "6800", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_retry_count": "3", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_bind_retry_delay": "5", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_client": "false", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_mds": "false", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_mgr": "false", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_mon": "false", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_blackhole_osd": "false", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_client_mode": "crc secure", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_cluster_mode": "crc secure", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_cluster_type": "", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_compress_secure": "false", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_connection_idle_timeout": "900", 2024-04-18T21:51:38.232 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_connection_ready_timeout": "10", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_crc_data": "true", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_crc_header": "true", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_bad_msg": "false", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_bug": "true", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_old_message": "true", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_skipped_message": "false", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_coremask": "0xF", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_enable_tso": "true", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-18T21:51:38.233 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_hugepages": "", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_lro": "true", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_memory_channel": "4", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_pmd": "", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_port_id": "0", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_dump_on_send": "false", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_initial_backoff": "0.200000", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_delay_max": "1.000000", 2024-04-18T21:51:38.234 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_delay_type": "", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_network_congestion": "0", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_inject_socket_failures": "0", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_learn_addr_from_peer": "true", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_max_accept_failures": "4", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_max_backoff": "15.000000", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_mon_client_mode": "secure crc", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_mon_service_mode": "secure crc", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_osd_compress_min_size": "1024", 2024-04-18T21:51:38.235 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_osd_compress_mode": "none", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_pq_min_cost": "65536", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_public_type": "", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_service_mode": "crc secure", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_tcp_listen_backlog": "512", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_tcp_nodelay": "true", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_tcp_rcvbuf": "0", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "ms_type": "async+posix", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "no_config_file": "false", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_completion_locks_per_session": "32", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-18T21:51:38.236 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_inflight_ops": "1024", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_tick_interval": "5.000000", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter_timeout": "10.000000", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "objectstore_blackhole": "false", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "openssl_engine_opts": "", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_delay_time": "5.000000", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_hist_halflife": "1000", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_max_low_ops": "2", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_max_ops": "4", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-18T21:51:38.237 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_agent_slop": "0.020000", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_asio_thread_count": "2", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_async_recovery_min_cost": "1", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backfill_scan_max": "512", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backfill_scan_min": "64", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backoff_on_degraded": "false", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backoff_on_peering": "false", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_backoff_on_unfound": "true", 2024-04-18T21:51:38.238 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_beacon_report_interval": "300", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bench_duration": "30", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bench_max_block_size": "67108864", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_blkin_trace_all": "false", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_check_for_log_corruption": "false", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_class_update_on_start": "true", 2024-04-18T21:51:38.239 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_client_message_cap": "256", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_client_message_size_cap": "524288000", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_client_op_priority": "63", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_client_watch_timeout": "30", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_command_max_records": "256", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_compact_on_start": "false", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_crush_update_on_start": "true", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_crush_update_weight_set": "true", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-18T21:51:38.240 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_feed_pullee": "-1", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_inject_dispatch_delay_probability": "0.100000", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_misdirected_ops": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_no_acting_change": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_no_purge_strays": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_op_order": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_shutdown": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-18T21:51:38.241 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_trim_objects": "false", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_verify_snaps": "false", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_keys": "1024", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_stride": "524288", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-18T21:51:38.242 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_default_notify_timeout": "30", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_delete_sleep": "0.000000", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_discard_disconnected_ops": "true", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_enable_op_tracker": "true", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_info": "true", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_shutdown": "true", 2024-04-18T21:51:38.243 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_function_tracing": "false", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_grace": "20", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_interval": "6", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_min_peers": "10", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_min_size": "2000", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_stale": "600", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_hit_set_max_size": "100000", 2024-04-18T21:51:38.244 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_hit_set_min_size": "1000", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_journal_size": "5120", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_kill_backfill_at": "0", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_cache_size": "50", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_dedup": "true", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_message_max": "40", 2024-04-18T21:51:38.245 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_map_share_max_epochs": "40", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_attr_name_len": "100", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_attr_size": "0", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_backfills": "1", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_markdown_count": "5", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_markdown_period": "600", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_object_name_len": "2048", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_object_namespace_len": "256", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_object_size": "134217728", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_pg_blocked_by": "16", 2024-04-18T21:51:38.246 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_pg_log_entries": "10000", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_pgls": "1024", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_push_cost": "8388608", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_push_objects": "10", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_scrubs": "3", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_trimming_pgs": "2", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_write_op_reply_len": "64", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_max_write_size": "90", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-18T21:51:38.247 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_profile": "balanced", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-18T21:51:38.248 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_base": "805306368", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_cache_min": "134217728", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_target": "4294967296", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_target_autotune": "false", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_min_pg_log_entries": "250", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_min_recovery_priority": "0", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-18T21:51:38.249 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_report_interval": "5", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_num_cache_shards": "32", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_num_op_tracker_shard": "32", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_numa_auto_affinity": "true", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_numa_node": "-1", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_numa_prefer_iface": "true", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_objecter_finishers": "1", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_objectstore": "bluestore", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_objectstore_fuse": "false", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_objectstore_tracing": "false", 2024-04-18T21:51:38.250 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_complaint_time": "30.000000", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_history_duration": "600", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_history_size": "20", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_history_slow_op_size": "20", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_log_threshold": "5", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_shards": "0", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_shards_hdd": "5", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_shards_ssd": "8", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_pq_min_cost": "65536", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-18T21:51:38.251 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_queue_cut_off": "high", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_op_thread_timeout": "15", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_open_classes_on_start": "true", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_os_flags": "0", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_peering_op_priority": "255", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_delete_cost": "1048576", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_delete_priority": "5", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_log_trim_max": "10000", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_log_trim_min": "100", 2024-04-18T21:51:38.252 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_crimson": "false", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-18T21:51:38.253 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_flags": "0", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_min_size": "0", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_pg_num": "32", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_pgp_num": "0", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_read_ratio": "70", 2024-04-18T21:51:38.254 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_size": "2", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_default_type": "replicated", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_push_per_object_cost": "1000", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recover_clone_overlap": "true", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_cost": "20971520", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_active": "0", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-18T21:51:38.255 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_max_single_start": "1", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_op_priority": "3", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_priority": "5", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_sleep": "0.000000", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_repair_during_recovery": "false", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_requested_scrub_priority": "120", 2024-04-18T21:51:38.256 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_auto_repair": "false", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_begin_hour": "0", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_begin_week_day": "0", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_chunk_max": "25", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_chunk_min": "5", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_cost": "52428800", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_disable_reservation_queuing": "true", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_during_recovery": "false", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_end_hour": "0", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_end_week_day": "0", 2024-04-18T21:51:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_event_cost": "4096", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_invalid_stats": "true", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_max_preemptions": "5", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_priority": "5", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_sleep": "0.000000", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-18T21:51:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_skip_data_digest": "false", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_smart_report_timeout": "5", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_cost": "1048576", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_priority": "5", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-18T21:51:38.259 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_target_transaction_size": "30", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_tracing": "false", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_use_stale_snap": "false", 2024-04-18T21:51:38.260 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "osdc_blkin_trace_all": "false", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_kill_at": "0", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_max_join_drift": "10", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_min": "500", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_min_wait": "0.050000", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_propose_interval": "1.000000", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_service_trim_max": "500", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_service_trim_min": "250", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_stash_full_interval": "25", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_trim_max": "500", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "paxos_trim_min": "250", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "perf": "true", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-18T21:51:38.261 INFO:teuthology.orchestra.run.smithi097.stdout: "plugin_dir": "/usr/lib/ceph", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "-", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrv": "", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "public_bind_addr": "-", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "public_network": "", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "public_network_interface": "", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "qat_compressor_enabled": "false", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "qat_compressor_session_max_number": "256", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "rados_mon_op_timeout": "0", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "rados_osd_op_timeout": "0", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "rados_tracing": "false", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_atime_update_interval": "60", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-18T21:51:38.262 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_balance_parent_reads": "false", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_balance_snap_reads": "false", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_blkin_trace_all": "false", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache": "true", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_policy": "writearound", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_size": "33554432", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-18T21:51:38.263 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_clone_copy_on_read": "false", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_compression_hint": "none", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_concurrent_management_ops": "10", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_clone_format": "auto", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_data_pool": "", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_features": "61", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_format": "2", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_map_options": "", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_order": "22", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_pool": "rbd", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_stripe_count": "0", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_default_stripe_unit": "0", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-18T21:51:38.264 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_enable_alloc_hint": "true", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_io_scheduler": "simple", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_order": "24", 2024-04-18T21:51:38.265 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_pool": "", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_journal_splay_width": "4", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_localize_parent_reads": "false", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_localize_snap_reads": "false", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-18T21:51:38.266 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-18T21:51:38.267 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_mtime_update_interval": "60", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_non_blocking_aio": "true", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_op_thread_timeout": "60", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_op_threads": "1", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_parent_cache_enabled": "false", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-18T21:51:38.268 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_plugins": "", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_bps_burst": "0", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_bps_limit": "0", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_exclude_ops": "0", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_iops_burst": "0", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_iops_limit": "0", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-18T21:51:38.269 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_read_from_replica_policy": "default", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-18T21:51:38.270 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_skip_partial_discard": "true", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_tracing": "false", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_validate_names": "true", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_validate_pool": "true", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "restapi_base_url": "", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "restapi_log_level": "", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_acl_grants_max_num": "100", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_admin_entry": "admin", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_backend_store": "rados", 2024-04-18T21:51:38.271 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_barbican_url": "", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_beast_enable_async": "true", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_counters_cache": "false", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cache_enabled": "true", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cache_expiry_interval": "900", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cache_lru_size": "10000", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_config_store": "rados", 2024-04-18T21:51:38.272 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_content_length_compat": "false", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_copy_obj_progress": "true", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cors_rules_max_num": "100", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_cross_domain_policy": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_addr": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_password": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_kmip_username": "", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_require_ssl": "true", 2024-04-18T21:51:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-18T21:51:38.274 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_addr": "", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_auth": "token", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_namespace": "", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_prefix": "", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_token_file": "", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_buffersize": "524288", 2024-04-18T21:51:38.275 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_low_speed_time": "300", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-18T21:51:38.276 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_d4n_port": "6379", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_log_changes_size": "1000", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_log_num_shards": "128", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_log_window": "30", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-18T21:51:38.277 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-18T21:51:38.278 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dns_name": "", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dns_s3website_name": "", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_dynamic_resharding": "true", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_gc_threads": "true", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_lc_threads": "true", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_ops_log": "false", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_quota_threads": "true", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_static_website": "false", 2024-04-18T21:51:38.279 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enable_usage_log": "false", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_enforce_swift_acls": "true", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_exit_timeout_secs": "120", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_expose_bucket": "false", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_extended_http_attrs": "", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_filter": "none", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_frontends": "beast port=7480", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_deferred": "50", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_objs": "32", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-18T21:51:38.280 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_gc_processor_period": "3600", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_init_timeout": "300", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_kafka_connection_idle": "300", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-18T21:51:38.281 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_domain": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_password": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_password_path": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_project": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_tenant": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_token": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_token_path": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_admin_user": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_api_version": "2", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_domain": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_password": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_project": "", 2024-04-18T21:51:38.282 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_barbican_user": "", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_url": "", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_debug_interval": "-1", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_lock_max_time": "90", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_max_objs": "32", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_max_rules": "1000", 2024-04-18T21:51:38.283 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_max_worker": "3", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lc_thread_delay": "0", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_dnattr": "uid", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_searchfilter": "", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ldap_uri": "", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_log_http_headers": "", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-18T21:51:38.284 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_log_object_name_utc": "false", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_attr_name_len": "0", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_attr_size": "0", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_chunk_size": "4194304", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_listing_results": "1000", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_notify_retries": "10", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_put_param_size": "1048576", 2024-04-18T21:51:38.285 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_put_size": "5368709120", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_max_slo_entries": "1000", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_md_log_max_shards": "64", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_mp_lock_max_time": "600", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-18T21:51:38.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_max_gc": "300", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_num_async_rados_threads": "32", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_num_control_oids": "8", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_numa_node": "-1", 2024-04-18T21:51:38.287 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_objexp_chunk_size": "100", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_objexp_gc_interval": "600", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_op_thread_timeout": "600", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_op_tracing": "false", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_opa_token": "", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_opa_url": "", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_opa_verify_ssl": "true", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-18T21:51:38.288 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ops_log_rados": "false", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_ops_log_socket_path": "", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_period_push_interval": "2.000000", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_cache_lanes": "3", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-18T21:51:38.289 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_cache_partitions": "3", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_print_continue": "true", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_rados_tracing": "false", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_realm": "", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_realm_id": "", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-18T21:51:38.290 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_region": "", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_request_uri": "", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_batch_size": "64", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_max_aio": "128", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_num_logs": "16", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_reshard_thread_interval": "600", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_resolve_cname": "false", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_run_sync_thread": "true", 2024-04-18T21:51:38.291 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_scheduler_type": "throttler", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_script_uri": "", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_service_provider_name": "", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-18T21:51:38.292 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_client_id": "", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_client_secret": "", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_entry": "sts", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_key": "sts", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_min_session_duration": "900", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sts_token_introspection_url": "", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_account_in_url": "false", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_auth_entry": "auth", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_auth_url": "", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_custom_header": "", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_need_stats": "true", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_tenant_name": "", 2024-04-18T21:51:38.293 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_token_expiration": "86400", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_url": "", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_url_prefix": "swift", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_lease_period": "120", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-18T21:51:38.294 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_thread_pool_size": "512", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_comment": "", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_createby": "", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_encoding": "", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_flag": "false", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_origin": "", 2024-04-18T21:51:38.295 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_torrent_tracker": "", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_trust_forwarded_https": "false", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_usage_max_shards": "32", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_usage_max_user_shards": "1", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_use_opa_authz": "false", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_counters_cache": "false", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_max_buckets": "1000", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_policies_max_num": "100", 2024-04-18T21:51:38.296 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_user_unique_email": "true", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_verify_ssl": "true", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zone": "", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zone_id": "", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zonegroup": "", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zonegroup_id": "", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgwlc_auto_session_clear": "true", 2024-04-18T21:51:38.297 INFO:teuthology.orchestra.run.smithi097.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_block_size": "4096", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_size": "536870912", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-18T21:51:38.298 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_index_type": "binary_search", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_partition_filters": "false", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_perf": "false", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "run_dir": "/var/run/ceph", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_block_create": "true", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_cache_lru_size": "67108864", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-18T21:51:38.299 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_default_max_object_size": "16777216", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_device_size": "53687091200", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_journal_batch_capacity": "16", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_main_device_type": "SSD", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-18T21:51:38.300 INFO:teuthology.orchestra.run.smithi097.stdout: "seastore_segment_size": "67108864", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "set_keepcaps": "false", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "setgroup": "", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "setuser": "", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "setuser_match_path": "", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "thp": "false", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "threadpool_default_timeout": "60", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout: "throttler_perf_counter": "true" 2024-04-18T21:51:38.301 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:38.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-18T21:51:38.443 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a: {} 2024-04-18T21:51:38.443 INFO:teuthology.orchestra.run.smithi097.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-18T21:51:38.457 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b: {} 2024-04-18T21:51:38.457 INFO:teuthology.orchestra.run.smithi097.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-18T21:51:38.473 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c: {} 2024-04-18T21:51:38.473 INFO:teuthology.orchestra.run.smithi097.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-18T21:51:38.490 DEBUG:teuthology.orchestra.run.smithi097:> type ceph-objectstore-tool 2024-04-18T21:51:38.494 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-18T21:51:38.495 DEBUG:teuthology.orchestra.run.smithi183:> type ceph-objectstore-tool 2024-04-18T21:51:38.500 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-18T21:51:38.500 DEBUG:teuthology.orchestra.run.smithi104:> type ceph-objectstore-tool 2024-04-18T21:51:38.506 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-18T21:51:38.506 INFO:teuthology.run_tasks:Running task rados... 2024-04-18T21:51:38.517 INFO:tasks.rados:Beginning rados... 2024-04-18T21:51:38.517 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-18T21:51:38.539 INFO:tasks.rados:joining rados 2024-04-18T21:51:38.539 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-18T21:51:38.539 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:51:38.539 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:51:38.539 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-18T21:51:38.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-18T21:51:38.540 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-18T21:51:38.540 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-18T21:51:38.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:51:38.541 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-18T21:51:38.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:51:38.541 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-18T21:51:38.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-18T21:51:38.542 INFO:tasks.rados:clients are ['client.0'] 2024-04-18T21:51:38.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lrcprofile crush-failure-domain=osd k=4 l=3 m=2 name=lrcprofile plugin=lrc 2024-04-18T21:51:38.548 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:38.547+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:38.642 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:38.639+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:38.692 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:51:38.692 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:51:38.710 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:51:38.710 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:51:38.718 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:38.718 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:38.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:51:38.742 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:51:38.742 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:51:38.743 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:38.739+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:38.773 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:51:38.773 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:51:38.793 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:51:38.793 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:51:38.813 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:51:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:51:38.834 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:51:38.834 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:51:38.844 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:38.839+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:38.893 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:51:38.894 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:51:38.894 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:38.894 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:38.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 23, 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout: "fsid": "9a7b119f-53a7-4e46-8d6e-229eac84ebee", 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout: "created": "2024-04-18T21:51:17.630312+0000", 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout: "modified": "2024-04-18T21:51:33.804996+0000", 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout: "last_up_change": "2024-04-18T21:51:29.769135+0000", 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout: "last_in_change": "2024-04-18T21:51:21.247402+0000", 2024-04-18T21:51:38.913 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_num": 5799936, 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_set": [ 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "sortbitwise" 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_version": 5, 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "full_ratio": 0.94999998807907104, 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-18T21:51:38.914 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_snapshot": "", 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_max": 2, 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "max_osd": 12, 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "require_min_compat_client": "luminous", 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "min_compat_client": "jewel", 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "require_osd_release": "squid", 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_crimson": false, 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "pools": [ 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 1, 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_name": ".mgr", 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "create_time": "2024-04-18T21:51:28.321879+0000", 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_names": "hashpspool", 2024-04-18T21:51:38.915 INFO:teuthology.orchestra.run.smithi097.stdout: "type": 1, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 2, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "min_size": 1, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_rule": 0, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "object_hash": 2, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscale_mode": "off", 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num": 1, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num": 1, 2024-04-18T21:51:38.916 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num_target": 1, 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_target": 1, 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_pending": 1, 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "last_pg_merge_meta": { 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "source_pgid": "0.0", 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "ready_epoch": 0, 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_started": 0, 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_clean": 0, 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "source_version": "0'0", 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "target_version": "0'0" 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "last_change": "20", 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend": "0", 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-18T21:51:38.917 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_preluminous": "0", 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "auid": 0, 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_mode": "selfmanaged", 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_seq": 0, 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_epoch": 0, 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_snaps": [], 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "tiers": [], 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "tier_of": -1, 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "read_tier": -1, 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "write_tier": -1, 2024-04-18T21:51:38.918 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_mode": "none", 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_bytes": 0, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_objects": 0, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_flush_age": 0, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_evict_age": 0, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_profile": "", 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_params": { 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "none" 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_period": 0, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_count": 0, 2024-04-18T21:51:38.919 INFO:teuthology.orchestra.run.smithi097.stdout: "use_gmt_hitset": true, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "min_read_recency_for_promote": 0, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "grade_table": [], 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "stripe_width": 0, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "expected_num_objects": 0, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "fast_read": false, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "options": { 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_max": 32, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_min": 1 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.920 INFO:teuthology.orchestra.run.smithi097.stdout: "application_metadata": { 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": {} 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "read_balance": { 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "score_type": "Fair distribution", 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "score_acting": 11.760000228881836, 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "score_stable": 11.760000228881836, 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "optimal_score": 0.17000000178813934, 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "raw_score_acting": 2, 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "raw_score_stable": 2, 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "average_primary_affinity": 1, 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: "average_primary_affinity_weighted": 1 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 2, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_name": "rbd", 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "create_time": "2024-04-18T21:51:30.835497+0000", 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_names": "hashpspool", 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "type": 1, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 2, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "min_size": 1, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_rule": 0, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2024-04-18T21:51:38.922 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "object_hash": 2, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscale_mode": "off", 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num": 8, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num": 8, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num_target": 8, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_target": 8, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_pending": 8, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "last_pg_merge_meta": { 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "source_pgid": "0.0", 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "ready_epoch": 0, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_started": 0, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_clean": 0, 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "source_version": "0'0", 2024-04-18T21:51:38.923 INFO:teuthology.orchestra.run.smithi097.stdout: "target_version": "0'0" 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "last_change": "23", 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend": "0", 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_preluminous": "0", 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "auid": 0, 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_mode": "selfmanaged", 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_seq": 0, 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_epoch": 0, 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_snaps": [], 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2024-04-18T21:51:38.924 INFO:teuthology.orchestra.run.smithi097.stdout: "tiers": [], 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "tier_of": -1, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "read_tier": -1, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "write_tier": -1, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_mode": "none", 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_bytes": 0, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_objects": 0, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_flush_age": 0, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_evict_age": 0, 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_profile": "", 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_params": { 2024-04-18T21:51:38.925 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "none" 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_period": 0, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_count": 0, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "use_gmt_hitset": true, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "min_read_recency_for_promote": 0, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "grade_table": [], 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "stripe_width": 0, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "expected_num_objects": 0, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "fast_read": false, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "options": {}, 2024-04-18T21:51:38.926 INFO:teuthology.orchestra.run.smithi097.stdout: "application_metadata": { 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd": {} 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "read_balance": { 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "score_type": "Fair distribution", 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "score_acting": 3.0099999904632568, 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "score_stable": 3.0099999904632568, 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "optimal_score": 0.82999998331069946, 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "raw_score_acting": 2.5, 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "raw_score_stable": 2.5, 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "average_primary_affinity": 1, 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: "average_primary_affinity_weighted": 1 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.927 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 0, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "2a27387f-73d5-4bf2-bb86-80c069a1d6f8", 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 0, 2024-04-18T21:51:38.928 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6808", 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6812", 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.929 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6817", 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6820", 2024-04-18T21:51:38.930 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6830", 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6831", 2024-04-18T21:51:38.931 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6824", 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6828", 2024-04-18T21:51:38.932 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.97:6812/3458443931", 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.97:6820/3458443931", 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.97:6831/3458443931", 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.97:6828/3458443931", 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.933 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 1, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "2f6c823f-4b24-4ecf-9561-710723acded2", 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 0, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.934 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6800", 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6801", 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.935 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6802", 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6803", 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.936 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6811", 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6815", 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6805", 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6807", 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.938 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.97:6801/1179456160", 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.97:6803/1179456160", 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.97:6815/1179456160", 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.97:6807/1179456160", 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 2, 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "05866927-b7ea-44e6-8a5e-f23263011fa4", 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.939 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 0, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.940 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6809", 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6813", 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.941 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6816", 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6819", 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.942 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6825", 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6827", 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.943 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6821", 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6823", 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.97:6813/598211345", 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.97:6819/598211345", 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.97:6827/598211345", 2024-04-18T21:51:38.944 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.97:6823/598211345", 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 3, 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "1fbdd117-432a-4bef-879e-96823b61820a", 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.945 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 21, 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6804", 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.946 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6806", 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6810", 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.947 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6814", 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6826", 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.948 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6829", 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6818", 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.949 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6822", 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.97:6806/40393780", 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.97:6814/40393780", 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.97:6829/40393780", 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.97:6822/40393780", 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 4, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "a7f8da99-45f2-4d8f-8c8c-efd9632b0f61", 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 0, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.951 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6808", 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6809", 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6810", 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6811", 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6814", 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6815", 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.954 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6812", 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6813", 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:51:38.955 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.104:6809/3973913623", 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.104:6811/3973913623", 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.104:6815/3973913623", 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.104:6813/3973913623", 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.956 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 5, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "2230dfaa-7c85-469b-a95b-594960d77b28", 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 21, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6800", 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6801", 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.958 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6802", 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6803", 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.959 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6806", 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6807", 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6804", 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6805", 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.104:6801/1478992259", 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.104:6803/1478992259", 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.104:6807/1478992259", 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.104:6805/1478992259", 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 6, 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "e7437af9-b161-482e-8939-4d2b4c2da10e", 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.962 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 21, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.963 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6816", 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6817", 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.964 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6818", 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6819", 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.965 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6822", 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6823", 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.966 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6820", 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6821", 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.104:6817/462814955", 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.104:6819/462814955", 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.104:6823/462814955", 2024-04-18T21:51:38.967 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.104:6821/462814955", 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 7, 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "a062c3b3-7e00-4bdf-9731-0072826522c9", 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 21, 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6824", 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.969 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6825", 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6826", 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6827", 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6830", 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6831", 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6828", 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6829", 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.104:6825/581913590", 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.104:6827/581913590", 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.104:6831/581913590", 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.104:6829/581913590", 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 8, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "f2473c34-2db3-4806-a6ea-bb46f42c4ab8", 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 17, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 0, 2024-04-18T21:51:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6804", 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6806", 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:51:38.975 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6808", 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6812", 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:51:38.976 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6822", 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6825", 2024-04-18T21:51:38.977 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6816", 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6819", 2024-04-18T21:51:38.978 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.183:6806/1020504720", 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.183:6812/1020504720", 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.183:6825/1020504720", 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.183:6819/1020504720", 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.979 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 9, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "f08d755f-cc20-468c-b3ca-cfb88b44bdf1", 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 21, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6810", 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6814", 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6818", 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6821", 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6829", 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6830", 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6824", 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6827", 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.183:6814/1194329140", 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.183:6821/1194329140", 2024-04-18T21:51:38.984 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.183:6830/1194329140", 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.183:6827/1194329140", 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 10, 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "247fad9b-8847-416f-bfbb-9d8b466c3aa6", 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.985 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 0, 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6809", 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:51:38.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6813", 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6817", 2024-04-18T21:51:38.987 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6820", 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6828", 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:51:38.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6831", 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6823", 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6826", 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.183:6813/1192340806", 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.183:6820/1192340806", 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.183:6831/1192340806", 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.183:6826/1192340806", 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.990 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 11, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "4a51c1be-4125-451a-a910-c8e364fe21be", 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 17, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 21, 2024-04-18T21:51:38.991 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6800", 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6801", 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:51:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6802", 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6803", 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:51:38.993 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6811", 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6815", 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:51:38.994 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6805", 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6807", 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:51:38.995 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.183:6801/3144695911", 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.183:6803/3144695911", 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.183:6815/3144695911", 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.183:6807/3144695911", 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_xinfo": [ 2024-04-18T21:51:38.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 0, 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.945475+0000", 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 1, 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:38.997 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.965087+0000", 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 2, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:38.998 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.895463+0000", 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 3, 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.826302+0000", 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:38.999 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 4, 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.817690+0000", 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 5, 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:39.000 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.671915+0000", 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 6, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:39.001 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.835011+0000", 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 7, 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.809799+0000", 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:39.002 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 8, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.710699+0000", 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 9, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:39.003 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.754860+0000", 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 10, 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.849759+0000", 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:39.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 11, 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.728380+0000", 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_upmap": [], 2024-04-18T21:51:39.005 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_upmap_items": [], 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_upmap_primaries": [], 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_temp": [], 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_temp": [], 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "blocklist": {}, 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "range_blocklist": {}, 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_profiles": { 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "default": { 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-failure-domain": "osd", 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "k": "2", 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "m": "1", 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "plugin": "jerasure", 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "technique": "reed_sol_van" 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:39.006 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps_queue": [], 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "new_removed_snaps": [], 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "new_purged_snaps": [], 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_node_flags": {}, 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "device_class_flags": {}, 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "stretch_mode": { 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "stretch_mode_enabled": false, 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "stretch_bucket_count": 0, 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "degraded_stretch_mode": 0, 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "recovering_stretch_mode": 0, 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: "stretch_mode_bucket": 0 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:39.007 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:39.008 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:51:39.009 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:51:39.010 INFO:tasks.thrashosds.thrasher:j is {'epoch': 23, 'fsid': '9a7b119f-53a7-4e46-8d6e-229eac84ebee', 'created': '2024-04-18T21:51:17.630312+0000', 'modified': '2024-04-18T21:51:33.804996+0000', 'last_up_change': '2024-04-18T21:51:29.769135+0000', 'last_in_change': '2024-04-18T21:51:21.247402+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T21:51:28.321879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-18T21:51:30.835497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '2a27387f-73d5-4bf2-bb86-80c069a1d6f8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3458443931}, {'type': 'v1', 'addr': '172.21.15.97:6812', 'nonce': 3458443931}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6817', 'nonce': 3458443931}, {'type': 'v1', 'addr': '172.21.15.97:6820', 'nonce': 3458443931}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 3458443931}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 3458443931}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 3458443931}, {'type': 'v1', 'addr': '172.21.15.97:6828', 'nonce': 3458443931}]}, 'public_addr': '172.21.15.97:6812/3458443931', 'cluster_addr': '172.21.15.97:6820/3458443931', 'heartbeat_back_addr': '172.21.15.97:6831/3458443931', 'heartbeat_front_addr': '172.21.15.97:6828/3458443931', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f6c823f-4b24-4ecf-9561-710723acded2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1179456160}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1179456160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1179456160}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1179456160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 1179456160}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 1179456160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 1179456160}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1179456160}]}, 'public_addr': '172.21.15.97:6801/1179456160', 'cluster_addr': '172.21.15.97:6803/1179456160', 'heartbeat_back_addr': '172.21.15.97:6815/1179456160', 'heartbeat_front_addr': '172.21.15.97:6807/1179456160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '05866927-b7ea-44e6-8a5e-f23263011fa4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 598211345}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 598211345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 598211345}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 598211345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6825', 'nonce': 598211345}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 598211345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6821', 'nonce': 598211345}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 598211345}]}, 'public_addr': '172.21.15.97:6813/598211345', 'cluster_addr': '172.21.15.97:6819/598211345', 'heartbeat_back_addr': '172.21.15.97:6827/598211345', 'heartbeat_front_addr': '172.21.15.97:6823/598211345', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1fbdd117-432a-4bef-879e-96823b61820a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 40393780}, {'type': 'v1', 'addr': '172.21.15.97:6806', 'nonce': 40393780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 40393780}, {'type': 'v1', 'addr': '172.21.15.97:6814', 'nonce': 40393780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 40393780}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 40393780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 40393780}, {'type': 'v1', 'addr': '172.21.15.97:6822', 'nonce': 40393780}]}, 'public_addr': '172.21.15.97:6806/40393780', 'cluster_addr': '172.21.15.97:6814/40393780', 'heartbeat_back_addr': '172.21.15.97:6829/40393780', 'heartbeat_front_addr': '172.21.15.97:6822/40393780', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a7f8da99-45f2-4d8f-8c8c-efd9632b0f61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6808', 'nonce': 3973913623}, {'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 3973913623}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6810', 'nonce': 3973913623}, {'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 3973913623}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6814', 'nonce': 3973913623}, {'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 3973913623}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6812', 'nonce': 3973913623}, {'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 3973913623}]}, 'public_addr': '172.21.15.104:6809/3973913623', 'cluster_addr': '172.21.15.104:6811/3973913623', 'heartbeat_back_addr': '172.21.15.104:6815/3973913623', 'heartbeat_front_addr': '172.21.15.104:6813/3973913623', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2230dfaa-7c85-469b-a95b-594960d77b28', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6800', 'nonce': 1478992259}, {'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1478992259}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6802', 'nonce': 1478992259}, {'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1478992259}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6806', 'nonce': 1478992259}, {'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 1478992259}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6804', 'nonce': 1478992259}, {'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 1478992259}]}, 'public_addr': '172.21.15.104:6801/1478992259', 'cluster_addr': '172.21.15.104:6803/1478992259', 'heartbeat_back_addr': '172.21.15.104:6807/1478992259', 'heartbeat_front_addr': '172.21.15.104:6805/1478992259', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7437af9-b161-482e-8939-4d2b4c2da10e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6816', 'nonce': 462814955}, {'type': 'v1', 'addr': '172.21.15.104:6817', 'nonce': 462814955}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6818', 'nonce': 462814955}, {'type': 'v1', 'addr': '172.21.15.104:6819', 'nonce': 462814955}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6822', 'nonce': 462814955}, {'type': 'v1', 'addr': '172.21.15.104:6823', 'nonce': 462814955}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6820', 'nonce': 462814955}, {'type': 'v1', 'addr': '172.21.15.104:6821', 'nonce': 462814955}]}, 'public_addr': '172.21.15.104:6817/462814955', 'cluster_addr': '172.21.15.104:6819/462814955', 'heartbeat_back_addr': '172.21.15.104:6823/462814955', 'heartbeat_front_addr': '172.21.15.104:6821/462814955', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a062c3b3-7e00-4bdf-9731-0072826522c9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6824', 'nonce': 581913590}, {'type': 'v1', 'addr': '172.21.15.104:6825', 'nonce': 581913590}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6826', 'nonce': 581913590}, {'type': 'v1', 'addr': '172.21.15.104:6827', 'nonce': 581913590}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6830', 'nonce': 581913590}, {'type': 'v1', 'addr': '172.21.15.104:6831', 'nonce': 581913590}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6828', 'nonce': 581913590}, {'type': 'v1', 'addr': '172.21.15.104:6829', 'nonce': 581913590}]}, 'public_addr': '172.21.15.104:6825/581913590', 'cluster_addr': '172.21.15.104:6827/581913590', 'heartbeat_back_addr': '172.21.15.104:6831/581913590', 'heartbeat_front_addr': '172.21.15.104:6829/581913590', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f2473c34-2db3-4806-a6ea-bb46f42c4ab8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 1020504720}, {'type': 'v1', 'addr': '172.21.15.183:6806', 'nonce': 1020504720}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 1020504720}, {'type': 'v1', 'addr': '172.21.15.183:6812', 'nonce': 1020504720}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 1020504720}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 1020504720}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 1020504720}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 1020504720}]}, 'public_addr': '172.21.15.183:6806/1020504720', 'cluster_addr': '172.21.15.183:6812/1020504720', 'heartbeat_back_addr': '172.21.15.183:6825/1020504720', 'heartbeat_front_addr': '172.21.15.183:6819/1020504720', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'f08d755f-cc20-468c-b3ca-cfb88b44bdf1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 1194329140}, {'type': 'v1', 'addr': '172.21.15.183:6814', 'nonce': 1194329140}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 1194329140}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 1194329140}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6829', 'nonce': 1194329140}, {'type': 'v1', 'addr': '172.21.15.183:6830', 'nonce': 1194329140}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 1194329140}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 1194329140}]}, 'public_addr': '172.21.15.183:6814/1194329140', 'cluster_addr': '172.21.15.183:6821/1194329140', 'heartbeat_back_addr': '172.21.15.183:6830/1194329140', 'heartbeat_front_addr': '172.21.15.183:6827/1194329140', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '247fad9b-8847-416f-bfbb-9d8b466c3aa6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6809', 'nonce': 1192340806}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 1192340806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6817', 'nonce': 1192340806}, {'type': 'v1', 'addr': '172.21.15.183:6820', 'nonce': 1192340806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 1192340806}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 1192340806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6823', 'nonce': 1192340806}, {'type': 'v1', 'addr': '172.21.15.183:6826', 'nonce': 1192340806}]}, 'public_addr': '172.21.15.183:6813/1192340806', 'cluster_addr': '172.21.15.183:6820/1192340806', 'heartbeat_back_addr': '172.21.15.183:6831/1192340806', 'heartbeat_front_addr': '172.21.15.183:6826/1192340806', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '4a51c1be-4125-451a-a910-c8e364fe21be', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6800', 'nonce': 3144695911}, {'type': 'v1', 'addr': '172.21.15.183:6801', 'nonce': 3144695911}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 3144695911}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 3144695911}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6811', 'nonce': 3144695911}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3144695911}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6805', 'nonce': 3144695911}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 3144695911}]}, 'public_addr': '172.21.15.183:6801/3144695911', 'cluster_addr': '172.21.15.183:6803/3144695911', 'heartbeat_back_addr': '172.21.15.183:6815/3144695911', 'heartbeat_front_addr': '172.21.15.183:6807/3144695911', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.945475+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.965087+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.895463+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.826302+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.817690+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.671915+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.835011+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.809799+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.710699+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.754860+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.849759+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.728380+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-18T21:51:39.010 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-18T21:51:39.010 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:51:39.011 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:51:39.011 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:51:39.011 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:51:39.012 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:51:39.012 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:51:39.013 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:39.007+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:39.109 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:39.107+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:39.113 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:39.113 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:39.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:51:39.217 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:39.215+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:39.284 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:39.284 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:39.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:51:39.311 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:39.307+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:39.412 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:39.411+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:39.450 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:39.450 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:39.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:51:39.513 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:39.511+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:39.570 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:39.570 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:39.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:51:39.614 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:39.611+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:39.691 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:39.691 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:39.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:51:39.749 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:39.747+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:39.816 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:39.811+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:39.825 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:39.825 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:39.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:51:39.883 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is set 2024-04-18T21:51:39.908 INFO:tasks.rados:starting run 0 out of 1 2024-04-18T21:51:39.908 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-18T21:51:39.908 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lrcprofile 2024-04-18T21:51:39.954 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:39.951+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:39.993 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:39.993 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:40.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:51:40.018 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:40.015+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:40.118 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:40.120+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:40.166 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:40.166 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:40.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:51:40.221 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:40.220+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:40.295 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:40.295 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:40.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:51:40.320 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:40.319+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:40.422 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:40.419+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:40.425 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:40.425 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:40.438 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:51:40.523 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:40.524+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:40.571 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:40.571 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:40.585 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:51:40.628 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:40.627+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:40.712 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:40.712 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:40.724 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:40.719+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:40.726 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:51:40.826 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:40.823+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:40.897 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:40.897 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:40.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:51:40.927 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:40.928+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:41.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:51:41.028 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:41.028+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:41.034 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:41.034 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:41.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:51:41.128 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:41.128+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:41.168 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:51:41.168 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:51:41.172 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:41.172 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:41.185 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:51:41.185 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:51:41.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:51:41.201 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:51:41.201 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:51:41.218 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:51:41.218 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:51:41.228 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:41.223+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:41.235 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:51:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:51:41.252 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:51:41.252 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:51:41.270 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:51:41.271 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:51:41.290 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:51:41.291 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:51:41.309 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:41.310 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:41.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:41.310 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [] 2024-04-18T21:51:41.310 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:41.312 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:51:41.312 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:51:41.324 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:51:41.329 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:41.323+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:41.333 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:51:41.333 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:51:41.353 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:51:41.353 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:51:41.375 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:51:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:51:41.430 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:41.427+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:41.442 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:41.442 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:41.442 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:51:41.442 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:41.456 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:51:41.544 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:41.543+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:41.624 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:41.625 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:41.632 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:41.631+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:41.638 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:51:41.733 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:41.727+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:41.802 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [] 2024-04-18T21:51:41.802 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:41.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:51:41.860 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:41.859+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:41.900 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'unique_pool_0' created 2024-04-18T21:51:41.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-18T21:51:41.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-18T21:51:41.934 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:41.931+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:41.945 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop_reply(client.4165.0:165 1.0 e25/18)", 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.915550+0000", 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.031012524, 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031029151000000001, 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:41.946 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915550+0000", 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915550+0000", 2024-04-18T21:51:41.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915551+0000", 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2860000000000001e-06 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915556+0000", 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1769999999999997e-06 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915574+0000", 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8036000000000001e-05 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915586+0000", 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1907e-05 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:41.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.64025105900000001, 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.64030709399999997, 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:41.950 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 165 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306312+0000", 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306317+0000", 2024-04-18T21:51:41.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0699999999999997e-06 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306327+0000", 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7030000000000001e-06 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306349+0000", 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2160000000000001e-05 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:41.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419233+0000", 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112884159 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419499+0000", 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265865 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419851+0000", 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035253999999999998 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:41.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.421040+0000", 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011886380000000001 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 2 2024-04-18T21:51:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:41.960 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:51:42.035 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:42.036+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:42.128 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:42.129 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:42.129 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:42.129 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:42.129 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:42.135 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:42.131+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:42.142 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:51:42.236 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:42.231+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:42.263 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:42.263 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:42.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:42.263 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:42.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.263 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-18T21:51:42.263 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.460714611, 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58281355499999998, 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 1 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798268+0000", 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5959999999999998e-06 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798283+0000", 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5495999999999999e-05 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798322+0000", 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8899999999999997e-05 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798347+0000", 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5510000000000001e-05 2024-04-18T21:51:42.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886734+0000", 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088386079000000006 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886749+0000", 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5019e-05 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.078589+0000", 2024-04-18T21:51:42.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19184037900000001 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079052+0000", 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046354399999999998 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079551+0000", 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049870299999999997 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.083572+0000", 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040212609999999999 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.380910+0000", 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29733748900000001 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381009+0000", 2024-04-18T21:51:42.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9025999999999997e-05 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381079+0000", 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9553000000000003e-05 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.5239052770000008, 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40660126499999999, 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 33 2024-04-18T21:51:42.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735075+0000", 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4300000000000001e-07 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735081+0000", 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8669999999999999e-06 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735094+0000", 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2813000000000001e-05 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735102+0000", 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4770000000000003e-06 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.737810+0000", 2024-04-18T21:51:42.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027080089999999999 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.737953+0000", 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014291900000000001 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.738162+0000", 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020895500000000001 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.940717+0000", 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20255461299999999 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141518+0000", 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20080082699999999 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141607+0000", 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9262e-05 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141676+0000", 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8579999999999997e-05 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.5238644870000009, 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50698967299999997, 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 34 2024-04-18T21:51:42.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735116+0000", 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735123+0000", 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3629999999999996e-06 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735156+0000", 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3030000000000001e-05 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735167+0000", 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.13e-05 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939428+0000", 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20426096299999999 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939615+0000", 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018654199999999999 2024-04-18T21:51:42.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939841+0000", 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022621800000000001 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141747+0000", 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201905318 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.241958+0000", 2024-04-18T21:51:42.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100211063 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242041+0000", 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3300000000000005e-05 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242105+0000", 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3895999999999999e-05 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.5237894880000002, 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61088117099999995, 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 35 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.284 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735191+0000", 2024-04-18T21:51:42.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3699999999999999e-07 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735266+0000", 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4856000000000003e-05 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735279+0000", 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.36e-05 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735288+0000", 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0860000000000008e-06 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242298+0000", 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50700964999999998 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242568+0000", 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027023100000000001 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242974+0000", 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040533399999999998 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246576+0000", 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036026919999999998 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345792+0000", 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099215503999999996 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345953+0000", 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161189 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346071+0000", 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118192 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.289 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.5236769950000006, 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61108955600000003, 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 36 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.23e-07 2024-04-18T21:51:42.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735310+0000", 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7560000000000002e-06 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735323+0000", 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735337+0000", 2024-04-18T21:51:42.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4049999999999999e-05 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244243+0000", 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50890638099999996 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244422+0000", 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000178495 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244638+0000", 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216562 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.247950+0000", 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033121750000000001 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346232+0000", 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098281564000000002 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346324+0000", 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2042999999999999e-05 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346392+0000", 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8052999999999994e-05 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.295 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.5236279760000002, 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71224671699999997, 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 37 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0999999999999998e-07 2024-04-18T21:51:42.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735358+0000", 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2639999999999999e-06 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735367+0000", 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735375+0000", 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5360000000000004e-06 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245345+0000", 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50996983699999998 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245615+0000", 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027018099999999998 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246022+0000", 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000407394 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.248536+0000", 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025132330000000001 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447444+0000", 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198908419 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447532+0000", 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8010000000000006e-05 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447598+0000", 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6383000000000005e-05 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.301 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:35.364767+0000", 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.8942122579999996, 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.404415319, 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 72 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.302 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364767+0000", 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364767+0000", 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364768+0000", 2024-04-18T21:51:42.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1200000000000001e-06 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364844+0000", 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5809000000000006e-05 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364859+0000", 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4396000000000001e-05 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364869+0000", 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.062e-05 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.365865+0000", 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00099560899999999995 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.365983+0000", 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011834599999999999 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.366170+0000", 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000186505 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.369553+0000", 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033832979999999999 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.768981+0000", 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39942772799999998 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769099+0000", 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118225 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769183+0000", 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3663000000000002e-05 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.8940880260000004, 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.404608044, 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364893+0000", 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1429999999999999e-06 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364969+0000", 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6136000000000004e-05 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364981+0000", 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2099999999999999e-05 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465122+0000", 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10014120999999999 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465287+0000", 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000164499 2024-04-18T21:51:42.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465554+0000", 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000267194 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465927+0000", 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000373171 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.467760+0000", 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018333749999999999 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769354+0000", 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.301593271 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769435+0000", 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1433000000000005e-05 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769500+0000", 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4511999999999995e-05 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:42.314 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.8755972930000002, 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40557132600000001, 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 85 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:42.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383383+0000", 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1799999999999999e-06 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383389+0000", 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6030000000000004e-06 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383400+0000", 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.059e-05 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383410+0000", 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.039e-05 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.485426+0000", 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10201608199999999 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.485677+0000", 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025043800000000002 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486126+0000", 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00044914 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489043+0000", 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029174309999999998 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788775+0000", 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29973160999999998 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788881+0000", 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106443 2024-04-18T21:51:42.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788954+0000", 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2418999999999999e-05 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.8755575570000005, 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40574237800000001, 2024-04-18T21:51:42.320 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 86 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383423+0000", 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4700000000000001e-07 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383428+0000", 2024-04-18T21:51:42.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383437+0000", 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4630000000000007e-06 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383445+0000", 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.994e-06 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486731+0000", 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103285425 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486864+0000", 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013336899999999999 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487072+0000", 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020769500000000001 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489073+0000", 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020010010000000001 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789030+0000", 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29995712499999999 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789104+0000", 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4257000000000004e-05 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789164+0000", 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0291999999999998e-05 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.8755180039999999, 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40594649700000002, 2024-04-18T21:51:42.326 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 87 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:42.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5400000000000001e-07 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383535+0000", 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2879000000000005e-05 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383546+0000", 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1362999999999999e-05 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483681+0000", 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100134521 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487677+0000", 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039962119999999999 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487809+0000", 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013202900000000001 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.488004+0000", 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000195198 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.490235+0000", 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002230073 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789259+0000", 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29902419899999999 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789354+0000", 2024-04-18T21:51:42.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5443000000000001e-05 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789408+0000", 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3925999999999998e-05 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.333 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.775260877, 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40540477699999999, 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 88 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.334 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483720+0000", 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5859999999999999e-06 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483731+0000", 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1090000000000001e-05 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483753+0000", 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1426000000000001e-05 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483765+0000", 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2013e-05 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489152+0000", 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053867120000000001 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489297+0000", 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014540199999999999 2024-04-18T21:51:42.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489556+0000", 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025916099999999999 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.490845+0000", 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00128853 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.888963+0000", 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39811801499999999 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.889050+0000", 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7583000000000005e-05 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.889123+0000", 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3258999999999998e-05 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.6102274630000002, 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61732750999999997, 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.340 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 117 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648753+0000", 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648758+0000", 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648768+0000", 2024-04-18T21:51:42.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.414e-06 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648777+0000", 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.363e-06 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863479+0000", 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214701524 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863596+0000", 2024-04-18T21:51:42.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117039 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863810+0000", 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021416800000000001 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.864846+0000", 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001036319 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265892+0000", 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40104641899999999 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265995+0000", 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010209600000000001 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.266080+0000", 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5049000000000005e-05 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.345 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.95331622299999996, 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.509317829, 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.346 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 158 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305665+0000", 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5230000000000001e-06 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305786+0000", 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012069 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305805+0000", 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9836000000000001e-05 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305821+0000", 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6030000000000001e-05 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307593+0000", 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017715299999999999 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307838+0000", 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024551800000000002 2024-04-18T21:51:42.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.308174+0000", 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033593400000000003 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413731+0000", 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105556865 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814825+0000", 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40109329500000002 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814911+0000", 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6138999999999996e-05 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814981+0000", 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0469000000000006e-05 2024-04-18T21:51:42.351 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.953131167, 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50938204099999995, 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.352 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 159 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305849+0000", 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0559999999999999e-06 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305856+0000", 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9600000000000003e-06 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305867+0000", 2024-04-18T21:51:42.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0910000000000001e-05 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305879+0000", 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.147e-05 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309071+0000", 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031920859999999998 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309303+0000", 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023236200000000001 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309671+0000", 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036798000000000001 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416908+0000", 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107236578 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815085+0000", 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39817670700000002 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815165+0000", 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0196000000000003e-05 2024-04-18T21:51:42.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815230+0000", 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5735999999999995e-05 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.358 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.95308786400000001, 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50958771300000005, 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 160 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305898+0000", 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1359999999999996e-06 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305907+0000", 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5300000000000002e-06 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305915+0000", 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1869999999999997e-06 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310667+0000", 2024-04-18T21:51:42.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047516800000000003 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310895+0000", 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000227875 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.311257+0000", 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000361737 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416933+0000", 2024-04-18T21:51:42.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10567686699999999 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815344+0000", 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39841079899999998 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815421+0000", 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7229000000000002e-05 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815479+0000", 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7973000000000002e-05 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:42.365 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.95305320199999999, 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60955713600000005, 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 161 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.366 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305927+0000", 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306056+0000", 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012895300000000001 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306068+0000", 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306077+0000", 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9430000000000006e-06 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412179+0000", 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106101709 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412267+0000", 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8251999999999995e-05 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412388+0000", 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121136 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420472+0000", 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080833510000000008 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915141+0000", 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49466944899999998 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915356+0000", 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215305 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915484+0000", 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012713500000000001 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.95288937900000004, 2024-04-18T21:51:42.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61031149600000001, 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 162 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306091+0000", 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306097+0000", 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6300000000000003e-06 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306106+0000", 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2199999999999998e-06 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306116+0000", 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8870000000000004e-06 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413869+0000", 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107753534 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414135+0000", 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026615200000000002 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414518+0000", 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038217299999999997 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420492+0000", 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059740879999999998 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915940+0000", 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49544827600000002 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916143+0000", 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000202752 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916402+0000", 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025901100000000002 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.95285300900000003, 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61064147300000005, 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 163 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.06e-07 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306136+0000", 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6999999999999997e-06 2024-04-18T21:51:42.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306146+0000", 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8439999999999993e-06 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306154+0000", 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415593+0000", 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109439392 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415833+0000", 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000240101 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416223+0000", 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000389804 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420508+0000", 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042851699999999996 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916537+0000", 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49602917499999999 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916669+0000", 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179899999999999 2024-04-18T21:51:42.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916768+0000", 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8879e-05 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.95281447600000002, 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61096294100000004, 2024-04-18T21:51:42.384 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 164 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306166+0000", 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306275+0000", 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109749 2024-04-18T21:51:42.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306287+0000", 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1182999999999999e-05 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306298+0000", 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1487e-05 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417044+0000", 2024-04-18T21:51:42.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110745818 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417282+0000", 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000237712 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417665+0000", 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000383194 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:42.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420523+0000", 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028578010000000001 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916880+0000", 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49635731900000002 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917054+0000", 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017436200000000001 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:42.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:42.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917128+0000", 2024-04-18T21:51:42.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3693000000000001e-05 2024-04-18T21:51:42.390 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.390 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.390 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.390 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:42.390 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:42.390 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:42.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:51:42.392 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:42.392+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:42.493 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:42.487+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:42.525 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:42.525 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:42.525 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:51:42.525 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:42.539 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:51:42.593 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:42.591+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:42.656 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:51:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:51:42.657 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:42.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:51:42.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:42.691+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:42.787 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:42.787 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:51:42.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:51:42.787 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [] 2024-04-18T21:51:42.787 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:42.795 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:42.791+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:42.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:51:42.896 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:42.891+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:42.908 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is set 2024-04-18T21:51:42.908 INFO:teuthology.orchestra.run.smithi097.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-18T21:51:42.919 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:42.920 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:42.920 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:51:42.920 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:42.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:51:42.996 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:42.996+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:43.046 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:43.046 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:43.046 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:51:43.046 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:51:43.046 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:43.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:51:43.097 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:43.095+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:43.198 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:43.195+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:43.223 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:43.223 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:51:43.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:51:43.223 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [] 2024-04-18T21:51:43.224 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:43.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:51:43.299 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:43.299+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:43.353 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:43.353 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:43.354 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:51:43.354 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:43.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:51:43.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:51:43.437 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:43.435+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:43.483 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:43.483 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:43.483 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:51:43.483 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:51:43.483 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:43.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:51:43.501 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:43.495+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:43.545 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:51:43.545 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:51:43.561 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:51:43.561 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:51:43.578 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:51:43.579 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:51:43.595 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:51:43.595 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:51:43.601 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:43.604+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:43.603 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:43.603 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:51:43.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:51:43.603 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [] 2024-04-18T21:51:43.603 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:43.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:51:43.616 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:51:43.617 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:51:43.636 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:51:43.636 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:51:43.657 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:51:43.657 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:51:43.678 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:51:43.679 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:51:43.697 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:51:43.697 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:51:43.702 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:43.699+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:43.717 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:51:43.717 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:51:43.737 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:51:43.737 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:51:43.758 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:51:43.758 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:51:43.767 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:43.767 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:43.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:51:43.827 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:43.828+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:43.899 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:43.900 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:43.905 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:43.908+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:43.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:51:43.950 DEBUG:teuthology.orchestra.run.smithi097:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight read -> 100 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight delete -> 50 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight snap_create -> 50 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight snap_remove -> 50 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight rollback -> 50 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight setattr -> 25 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight rmattr -> 25 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight copy_from -> 50 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:adding op weight append -> 100 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:ceph version 19.0.0-2369-g1a6a3e74 (1a6a3e742fd81c6d70be70552499bff767652c32) squid (dev) 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout:Configuration: 2024-04-18T21:51:43.971 INFO:tasks.rados.rados.0.smithi097.stdout: Number of operations: 400 2024-04-18T21:51:43.972 INFO:tasks.rados.rados.0.smithi097.stdout: Number of objects: 50 2024-04-18T21:51:43.972 INFO:tasks.rados.rados.0.smithi097.stdout: Max in flight operations: 16 2024-04-18T21:51:43.972 INFO:tasks.rados.rados.0.smithi097.stdout: Object size (in bytes): 4000000 2024-04-18T21:51:43.972 INFO:tasks.rados.rados.0.smithi097.stdout: Write stride min: 400000 2024-04-18T21:51:43.972 INFO:tasks.rados.rados.0.smithi097.stdout: Write stride max: 800000 2024-04-18T21:51:44.006 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:44.003+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:44.011 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:51:44.011 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:51:44.011 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-18T21:51:44.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:51:44.077 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:44.078 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:44.091 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:51:44.107 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:44.103+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:44.208 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:44.203+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:44.215 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:44.215 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:44.229 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:51:44.309 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:44.307+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:44.317 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:51:44.317 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:51:44.334 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-18T21:51:44.248073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":89,"num_write":138,"num_write_kb":1897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":328412,"kb_used_data":3512,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132133668,"statfs":{"total":1159641169920,"available":1159304876032,"internally_reserved":0,"allocated":3596288,"data_stored":1901784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":331855165},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,10,14],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001235"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"23'99","reported_seq":135,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:39.470773+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:51:39.470773+0000","last_peered":"2024-04-18T21:51:39.470773+0000","last_clean":"2024-04-18T21:51:39.470773+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:51:39.470773+0000","last_undegraded":"2024-04-18T21:51:39.470773+0000","last_fullsized":"2024-04-18T21:51:39.470773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":89,"num_write":138,"num_write_kb":1897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625132+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:51:34.625132+0000","last_peered":"2024-04-18T21:51:34.625132+0000","last_clean":"2024-04-18T21:51:34.625132+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:51:34.625132+0000","last_undegraded":"2024-04-18T21:51:34.625132+0000","last_fullsized":"2024-04-18T21:51:34.625132+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:33.819238+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:51:33.819238+0000","last_peered":"2024-04-18T21:51:33.819238+0000","last_clean":"2024-04-18T21:51:33.819238+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:51:33.819238+0000","last_undegraded":"2024-04-18T21:51:33.819238+0000","last_fullsized":"2024-04-18T21:51:33.819238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-18T21:51:43.226749+0000","last_change":"2024-04-18T21:51:43.126316+0000","last_active":"2024-04-18T21:51:43.226749+0000","last_peered":"2024-04-18T21:51:43.226749+0000","last_clean":"2024-04-18T21:51:43.226749+0000","last_became_active":"2024-04-18T21:51:43.125387+0000","last_became_peered":"2024-04-18T21:51:43.125387+0000","last_unstale":"2024-04-18T21:51:43.226749+0000","last_undegraded":"2024-04-18T21:51:43.226749+0000","last_fullsized":"2024-04-18T21:51:43.226749+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:51.432523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.729353+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:51:34.729353+0000","last_peered":"2024-04-18T21:51:34.729353+0000","last_clean":"2024-04-18T21:51:34.729353+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:51:34.729353+0000","last_undegraded":"2024-04-18T21:51:34.729353+0000","last_fullsized":"2024-04-18T21:51:34.729353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625091+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:51:34.625091+0000","last_peered":"2024-04-18T21:51:34.625091+0000","last_clean":"2024-04-18T21:51:34.625091+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:51:34.625091+0000","last_undegraded":"2024-04-18T21:51:34.625091+0000","last_fullsized":"2024-04-18T21:51:34.625091+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-18T21:51:42.924550+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:51:42.924550+0000","last_peered":"2024-04-18T21:51:42.924550+0000","last_clean":"2024-04-18T21:51:42.924550+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:51:42.924550+0000","last_undegraded":"2024-04-18T21:51:42.924550+0000","last_fullsized":"2024-04-18T21:51:42.924550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T21:51:41.891788+0000","last_change":"2024-04-18T21:51:41.891788+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:41.891788+0000","last_undegraded":"2024-04-18T21:51:41.891788+0000","last_fullsized":"2024-04-18T21:51:41.891788+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.579433+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:51:34.579433+0000","last_peered":"2024-04-18T21:51:34.579433+0000","last_clean":"2024-04-18T21:51:34.579433+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:51:34.579433+0000","last_undegraded":"2024-04-18T21:51:34.579433+0000","last_fullsized":"2024-04-18T21:51:34.579433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-18T21:51:43.434346+0000","last_change":"2024-04-18T21:51:43.233639+0000","last_active":"2024-04-18T21:51:43.434346+0000","last_peered":"2024-04-18T21:51:43.434346+0000","last_clean":"2024-04-18T21:51:43.434346+0000","last_became_active":"2024-04-18T21:51:43.232873+0000","last_became_peered":"2024-04-18T21:51:43.232873+0000","last_unstale":"2024-04-18T21:51:43.434346+0000","last_undegraded":"2024-04-18T21:51:43.434346+0000","last_fullsized":"2024-04-18T21:51:43.434346+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:51.791338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.629243+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:51:34.629243+0000","last_peered":"2024-04-18T21:51:34.629243+0000","last_clean":"2024-04-18T21:51:34.629243+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:51:34.629243+0000","last_undegraded":"2024-04-18T21:51:34.629243+0000","last_fullsized":"2024-04-18T21:51:34.629243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-18T21:51:43.334577+0000","last_change":"2024-04-18T21:51:43.134895+0000","last_active":"2024-04-18T21:51:43.334577+0000","last_peered":"2024-04-18T21:51:43.334577+0000","last_clean":"2024-04-18T21:51:43.334577+0000","last_became_active":"2024-04-18T21:51:43.033391+0000","last_became_peered":"2024-04-18T21:51:43.033391+0000","last_unstale":"2024-04-18T21:51:43.334577+0000","last_undegraded":"2024-04-18T21:51:43.334577+0000","last_fullsized":"2024-04-18T21:51:43.334577+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:46.385490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-18T21:51:34.625428+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:51:34.625428+0000","last_peered":"2024-04-18T21:51:34.625428+0000","last_clean":"2024-04-18T21:51:34.625428+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:51:34.625428+0000","last_undegraded":"2024-04-18T21:51:34.625428+0000","last_fullsized":"2024-04-18T21:51:34.625428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":89,"num_write":138,"num_write_kb":1897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":873004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378631,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":91784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":91784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411333,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":630784,"data_stored":489568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,10,14],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411334,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":84052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411332,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":630784,"data_stored":489568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444038,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":117384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":117384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:51:44.336 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-18T21:51:44.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-18T21:51:44.391 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:44.391 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:44.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:51:44.410 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:44.412+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:44.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:44.507+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:44.565 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:44.566 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:44.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:51:44.612 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:44.611+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:44.713 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:44.711+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:44.748 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:44.748 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:44.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:51:44.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:44.811+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:44.885 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:44.885 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:44.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:51:44.915 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:44.911+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:44.921 INFO:tasks.rados.rados.0.smithi097.stdout:1: write initial oid 1 2024-04-18T21:51:44.921 INFO:tasks.rados.rados.0.smithi097.stdout:1: seq_num 0 ranges {0=32768} 2024-04-18T21:51:44.921 INFO:teuthology.orchestra.run.smithi097.stderr:set pool 3 pg_num to 26 2024-04-18T21:51:44.922 INFO:tasks.rados.rados.0.smithi097.stdout:1: writing smithi09731216-1 from 0 to 32768 tid 1 2024-04-18T21:51:44.922 INFO:tasks.rados.rados.0.smithi097.stdout:2: write initial oid 2 2024-04-18T21:51:44.922 INFO:tasks.rados.rados.0.smithi097.stdout:2: seq_num 1 ranges {0=32768} 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:2: writing smithi09731216-2 from 0 to 32768 tid 1 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:3: write initial oid 3 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:3: seq_num 2 ranges {0=49152} 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:3: writing smithi09731216-3 from 0 to 49152 tid 1 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:4: write initial oid 4 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:4: seq_num 3 ranges {0=65536} 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:4: writing smithi09731216-4 from 0 to 65536 tid 1 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:5: write initial oid 5 2024-04-18T21:51:44.923 INFO:tasks.rados.rados.0.smithi097.stdout:5: seq_num 4 ranges {0=81920} 2024-04-18T21:51:44.924 INFO:tasks.rados.rados.0.smithi097.stdout:5: writing smithi09731216-5 from 0 to 81920 tid 1 2024-04-18T21:51:44.924 INFO:tasks.rados.rados.0.smithi097.stdout:6: write initial oid 6 2024-04-18T21:51:44.924 INFO:tasks.rados.rados.0.smithi097.stdout:6: seq_num 5 ranges {0=98304} 2024-04-18T21:51:44.925 INFO:tasks.rados.rados.0.smithi097.stdout:6: writing smithi09731216-6 from 0 to 98304 tid 1 2024-04-18T21:51:44.926 INFO:tasks.rados.rados.0.smithi097.stdout:7: write initial oid 7 2024-04-18T21:51:44.926 INFO:tasks.rados.rados.0.smithi097.stdout:7: seq_num 6 ranges {0=114688} 2024-04-18T21:51:44.929 INFO:tasks.rados.rados.0.smithi097.stdout:7: writing smithi09731216-7 from 0 to 114688 tid 1 2024-04-18T21:51:44.929 INFO:tasks.rados.rados.0.smithi097.stdout:8: write initial oid 8 2024-04-18T21:51:44.930 INFO:tasks.rados.rados.0.smithi097.stdout:8: seq_num 7 ranges {0=131072} 2024-04-18T21:51:44.930 INFO:tasks.rados.rados.0.smithi097.stdout:8: writing smithi09731216-8 from 0 to 131072 tid 1 2024-04-18T21:51:44.930 INFO:tasks.rados.rados.0.smithi097.stdout:9: write initial oid 9 2024-04-18T21:51:44.930 INFO:tasks.rados.rados.0.smithi097.stdout:9: seq_num 8 ranges {0=147456} 2024-04-18T21:51:44.930 INFO:tasks.rados.rados.0.smithi097.stdout:9: writing smithi09731216-9 from 0 to 147456 tid 1 2024-04-18T21:51:44.930 INFO:tasks.rados.rados.0.smithi097.stdout:10: write initial oid 10 2024-04-18T21:51:44.931 INFO:tasks.rados.rados.0.smithi097.stdout:10: seq_num 9 ranges {0=163840} 2024-04-18T21:51:44.932 INFO:tasks.rados.rados.0.smithi097.stdout:10: writing smithi09731216-10 from 0 to 163840 tid 1 2024-04-18T21:51:44.932 INFO:tasks.rados.rados.0.smithi097.stdout:11: write initial oid 11 2024-04-18T21:51:44.932 INFO:tasks.rados.rados.0.smithi097.stdout:11: seq_num 10 ranges {0=180224} 2024-04-18T21:51:44.933 INFO:tasks.rados.rados.0.smithi097.stdout:11: writing smithi09731216-11 from 0 to 180224 tid 1 2024-04-18T21:51:44.933 INFO:tasks.rados.rados.0.smithi097.stdout:12: write initial oid 12 2024-04-18T21:51:44.933 INFO:tasks.rados.rados.0.smithi097.stdout:12: seq_num 11 ranges {0=196608} 2024-04-18T21:51:44.934 INFO:tasks.rados.rados.0.smithi097.stdout:12: writing smithi09731216-12 from 0 to 196608 tid 1 2024-04-18T21:51:44.934 INFO:tasks.rados.rados.0.smithi097.stdout:13: write initial oid 13 2024-04-18T21:51:44.934 INFO:tasks.rados.rados.0.smithi097.stdout:13: seq_num 12 ranges {0=212992} 2024-04-18T21:51:44.936 INFO:tasks.rados.rados.0.smithi097.stdout:13: writing smithi09731216-13 from 0 to 212992 tid 1 2024-04-18T21:51:44.936 INFO:tasks.rados.rados.0.smithi097.stdout:14: write initial oid 14 2024-04-18T21:51:44.936 INFO:tasks.rados.rados.0.smithi097.stdout:14: seq_num 13 ranges {0=229376} 2024-04-18T21:51:44.937 INFO:tasks.rados.rados.0.smithi097.stdout:14: writing smithi09731216-14 from 0 to 229376 tid 1 2024-04-18T21:51:44.937 INFO:tasks.rados.rados.0.smithi097.stdout:15: write initial oid 15 2024-04-18T21:51:44.938 INFO:tasks.rados.rados.0.smithi097.stdout:15: seq_num 14 ranges {0=245760} 2024-04-18T21:51:44.939 INFO:tasks.rados.rados.0.smithi097.stdout:15: writing smithi09731216-15 from 0 to 245760 tid 1 2024-04-18T21:51:44.939 INFO:tasks.rados.rados.0.smithi097.stdout:16: write initial oid 16 2024-04-18T21:51:44.939 INFO:tasks.rados.rados.0.smithi097.stdout:16: seq_num 15 ranges {0=262144} 2024-04-18T21:51:44.941 INFO:tasks.rados.rados.0.smithi097.stdout:16: writing smithi09731216-16 from 0 to 262144 tid 1 2024-04-18T21:51:44.941 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:44.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:51:45.016 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:45.011+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:45.030 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:45.030 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:45.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:51:45.133 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:45.131+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:45.203 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:45.203 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:45.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:51:45.217 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:45.215+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:45.228 INFO:tasks.rados.rados.0.smithi097.stdout:14: finishing write tid 1 to smithi09731216-14 2024-04-18T21:51:45.228 INFO:tasks.rados.rados.0.smithi097.stdout:14: finishing write tid 2 to smithi09731216-14 2024-04-18T21:51:45.243 INFO:tasks.rados.rados.0.smithi097.stdout:10: finishing write tid 1 to smithi09731216-10 2024-04-18T21:51:45.244 INFO:tasks.rados.rados.0.smithi097.stdout:10: finishing write tid 2 to smithi09731216-10 2024-04-18T21:51:45.250 INFO:tasks.rados.rados.0.smithi097.stdout:16: finishing write tid 1 to smithi09731216-16 2024-04-18T21:51:45.318 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:45.320+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:45.333 INFO:tasks.rados.rados.0.smithi097.stdout:8: finishing write tid 1 to smithi09731216-8 2024-04-18T21:51:45.334 INFO:tasks.rados.rados.0.smithi097.stdout:8: finishing write tid 2 to smithi09731216-8 2024-04-18T21:51:45.336 INFO:tasks.rados.rados.0.smithi097.stdout:2: finishing write tid 1 to smithi09731216-2 2024-04-18T21:51:45.336 INFO:tasks.rados.rados.0.smithi097.stdout:2: finishing write tid 2 to smithi09731216-2 2024-04-18T21:51:45.345 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:45.345 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:45.351 INFO:tasks.rados.rados.0.smithi097.stdout:16: finishing write tid 2 to smithi09731216-16 2024-04-18T21:51:45.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:51:45.433 INFO:tasks.rados.rados.0.smithi097.stdout:9: finishing write tid 1 to smithi09731216-9 2024-04-18T21:51:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:9: finishing write tid 2 to smithi09731216-9 2024-04-18T21:51:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:13: finishing write tid 1 to smithi09731216-13 2024-04-18T21:51:45.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:45.431+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:45.436 INFO:tasks.rados.rados.0.smithi097.stdout:7: finishing write tid 1 to smithi09731216-7 2024-04-18T21:51:45.444 INFO:tasks.rados.rados.0.smithi097.stdout:13: finishing write tid 2 to smithi09731216-13 2024-04-18T21:51:45.519 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:45.515+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:45.522 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:45.522 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:45.531 INFO:tasks.rados.rados.0.smithi097.stdout:4: finishing write tid 1 to smithi09731216-4 2024-04-18T21:51:45.536 INFO:tasks.rados.rados.0.smithi097.stdout:14: finishing write tid 3 to smithi09731216-14 2024-04-18T21:51:45.536 INFO:tasks.rados.rados.0.smithi097.stdout:14: oid 14 updating version 0 to 2 2024-04-18T21:51:45.536 INFO:tasks.rados.rados.0.smithi097.stdout:14: oid 14 version 2 is already newer than 1 2024-04-18T21:51:45.536 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-18T21:51:45.536 INFO:tasks.rados.rados.0.smithi097.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-18T21:51:45.536 INFO:tasks.rados.rados.0.smithi097.stdout:14: done (15 left) 2024-04-18T21:51:45.536 INFO:tasks.rados.rados.0.smithi097.stdout:17: write initial oid 17 2024-04-18T21:51:45.536 INFO:tasks.rados.rados.0.smithi097.stdout:17: seq_num 16 ranges {0=278528} 2024-04-18T21:51:45.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:51:45.538 INFO:tasks.rados.rados.0.smithi097.stdout:17: writing smithi09731216-17 from 0 to 278528 tid 1 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout:8: finishing write tid 3 to smithi09731216-8 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout:8: oid 8 updating version 0 to 1 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout:8: oid 8 updating version 1 to 2 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout:8: done (15 left) 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout:18: write initial oid 18 2024-04-18T21:51:45.539 INFO:tasks.rados.rados.0.smithi097.stdout:18: seq_num 17 ranges {0=294912} 2024-04-18T21:51:45.541 INFO:tasks.rados.rados.0.smithi097.stdout:18: writing smithi09731216-18 from 0 to 294912 tid 1 2024-04-18T21:51:45.541 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.541 INFO:tasks.rados.rados.0.smithi097.stdout:2: finishing write tid 3 to smithi09731216-2 2024-04-18T21:51:45.542 INFO:tasks.rados.rados.0.smithi097.stdout:2: oid 2 updating version 0 to 1 2024-04-18T21:51:45.542 INFO:tasks.rados.rados.0.smithi097.stdout:2: oid 2 updating version 1 to 2 2024-04-18T21:51:45.542 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-18T21:51:45.542 INFO:tasks.rados.rados.0.smithi097.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-18T21:51:45.542 INFO:tasks.rados.rados.0.smithi097.stdout:2: done (15 left) 2024-04-18T21:51:45.542 INFO:tasks.rados.rados.0.smithi097.stdout:19: write initial oid 19 2024-04-18T21:51:45.542 INFO:tasks.rados.rados.0.smithi097.stdout:19: seq_num 18 ranges {0=311296} 2024-04-18T21:51:45.544 INFO:tasks.rados.rados.0.smithi097.stdout:19: writing smithi09731216-19 from 0 to 311296 tid 1 2024-04-18T21:51:45.544 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.556 INFO:tasks.rados.rados.0.smithi097.stdout:16: finishing write tid 3 to smithi09731216-16 2024-04-18T21:51:45.556 INFO:tasks.rados.rados.0.smithi097.stdout:16: oid 16 updating version 0 to 1 2024-04-18T21:51:45.556 INFO:tasks.rados.rados.0.smithi097.stdout:16: oid 16 updating version 1 to 2 2024-04-18T21:51:45.556 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 16 v 2 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-18T21:51:45.556 INFO:tasks.rados.rados.0.smithi097.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-18T21:51:45.556 INFO:tasks.rados.rados.0.smithi097.stdout:16: done (15 left) 2024-04-18T21:51:45.556 INFO:tasks.rados.rados.0.smithi097.stdout:20: write initial oid 20 2024-04-18T21:51:45.556 INFO:tasks.rados.rados.0.smithi097.stdout:20: seq_num 19 ranges {0=327680} 2024-04-18T21:51:45.559 INFO:tasks.rados.rados.0.smithi097.stdout:20: writing smithi09731216-20 from 0 to 327680 tid 1 2024-04-18T21:51:45.559 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.620 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:45.620+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:45.631 INFO:tasks.rados.rados.0.smithi097.stdout:1: finishing write tid 1 to smithi09731216-1 2024-04-18T21:51:45.631 INFO:tasks.rados.rados.0.smithi097.stdout:1: finishing write tid 2 to smithi09731216-1 2024-04-18T21:51:45.631 INFO:tasks.rados.rados.0.smithi097.stdout:5: finishing write tid 1 to smithi09731216-5 2024-04-18T21:51:45.638 INFO:tasks.rados.rados.0.smithi097.stdout:10: finishing write tid 3 to smithi09731216-10 2024-04-18T21:51:45.638 INFO:tasks.rados.rados.0.smithi097.stdout:10: oid 10 updating version 0 to 1 2024-04-18T21:51:45.638 INFO:tasks.rados.rados.0.smithi097.stdout:10: oid 10 updating version 1 to 2 2024-04-18T21:51:45.638 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-18T21:51:45.638 INFO:tasks.rados.rados.0.smithi097.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-18T21:51:45.638 INFO:tasks.rados.rados.0.smithi097.stdout:10: done (15 left) 2024-04-18T21:51:45.638 INFO:tasks.rados.rados.0.smithi097.stdout:21: write initial oid 21 2024-04-18T21:51:45.638 INFO:tasks.rados.rados.0.smithi097.stdout:21: seq_num 20 ranges {0=344064} 2024-04-18T21:51:45.641 INFO:tasks.rados.rados.0.smithi097.stdout:21: writing smithi09731216-21 from 0 to 344064 tid 1 2024-04-18T21:51:45.641 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.650 INFO:tasks.rados.rados.0.smithi097.stdout:9: finishing write tid 3 to smithi09731216-9 2024-04-18T21:51:45.650 INFO:tasks.rados.rados.0.smithi097.stdout:9: oid 9 updating version 0 to 1 2024-04-18T21:51:45.651 INFO:tasks.rados.rados.0.smithi097.stdout:9: oid 9 updating version 1 to 2 2024-04-18T21:51:45.651 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-18T21:51:45.651 INFO:tasks.rados.rados.0.smithi097.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-18T21:51:45.651 INFO:tasks.rados.rados.0.smithi097.stdout:9: done (15 left) 2024-04-18T21:51:45.651 INFO:tasks.rados.rados.0.smithi097.stdout:22: write initial oid 22 2024-04-18T21:51:45.651 INFO:tasks.rados.rados.0.smithi097.stdout:22: seq_num 21 ranges {0=360448} 2024-04-18T21:51:45.653 INFO:tasks.rados.rados.0.smithi097.stdout:22: writing smithi09731216-22 from 0 to 360448 tid 1 2024-04-18T21:51:45.654 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.654 INFO:tasks.rados.rados.0.smithi097.stdout:11: finishing write tid 1 to smithi09731216-11 2024-04-18T21:51:45.670 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:45.670 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:45.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:51:45.722 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:45.724+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:45.731 INFO:tasks.rados.rados.0.smithi097.stdout:5: finishing write tid 2 to smithi09731216-5 2024-04-18T21:51:45.731 INFO:tasks.rados.rados.0.smithi097.stdout:4: finishing write tid 2 to smithi09731216-4 2024-04-18T21:51:45.734 INFO:tasks.rados.rados.0.smithi097.stdout:12: finishing write tid 1 to smithi09731216-12 2024-04-18T21:51:45.742 INFO:tasks.rados.rados.0.smithi097.stdout:11: finishing write tid 2 to smithi09731216-11 2024-04-18T21:51:45.747 INFO:tasks.rados.rados.0.smithi097.stdout:18: finishing write tid 1 to smithi09731216-18 2024-04-18T21:51:45.748 INFO:tasks.rados.rados.0.smithi097.stdout:13: finishing write tid 3 to smithi09731216-13 2024-04-18T21:51:45.748 INFO:tasks.rados.rados.0.smithi097.stdout:13: oid 13 updating version 0 to 1 2024-04-18T21:51:45.748 INFO:tasks.rados.rados.0.smithi097.stdout:13: oid 13 updating version 1 to 2 2024-04-18T21:51:45.748 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-18T21:51:45.748 INFO:tasks.rados.rados.0.smithi097.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-18T21:51:45.748 INFO:tasks.rados.rados.0.smithi097.stdout:13: done (15 left) 2024-04-18T21:51:45.749 INFO:tasks.rados.rados.0.smithi097.stdout:23: write initial oid 23 2024-04-18T21:51:45.749 INFO:tasks.rados.rados.0.smithi097.stdout:23: seq_num 22 ranges {0=376832} 2024-04-18T21:51:45.755 INFO:tasks.rados.rados.0.smithi097.stdout:23: writing smithi09731216-23 from 0 to 376832 tid 1 2024-04-18T21:51:45.755 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:51:45.823 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:45.819+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:45.835 INFO:tasks.rados.rados.0.smithi097.stdout:12: finishing write tid 2 to smithi09731216-12 2024-04-18T21:51:45.846 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:45.846 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:45.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:51:45.864 INFO:tasks.rados.rados.0.smithi097.stdout:3: finishing write tid 1 to smithi09731216-3 2024-04-18T21:51:45.923 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:45.924+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:45.924 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:51:45.924 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:51:45.932 INFO:tasks.rados.rados.0.smithi097.stdout:7: finishing write tid 2 to smithi09731216-7 2024-04-18T21:51:45.933 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:51:45.935 INFO:tasks.rados.rados.0.smithi097.stdout:18: finishing write tid 2 to smithi09731216-18 2024-04-18T21:51:45.939 INFO:tasks.rados.rados.0.smithi097.stdout:1: finishing write tid 3 to smithi09731216-1 2024-04-18T21:51:45.939 INFO:tasks.rados.rados.0.smithi097.stdout:1: oid 1 updating version 0 to 2 2024-04-18T21:51:45.939 INFO:tasks.rados.rados.0.smithi097.stdout:1: oid 1 version 2 is already newer than 1 2024-04-18T21:51:45.939 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-18T21:51:45.939 INFO:tasks.rados.rados.0.smithi097.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:51:45.939 INFO:tasks.rados.rados.0.smithi097.stdout:1: done (15 left) 2024-04-18T21:51:45.939 INFO:tasks.rados.rados.0.smithi097.stdout:24: write initial oid 24 2024-04-18T21:51:45.939 INFO:tasks.rados.rados.0.smithi097.stdout:24: seq_num 23 ranges {0=393216} 2024-04-18T21:51:45.942 INFO:tasks.rados.rados.0.smithi097.stdout:24: writing smithi09731216-24 from 0 to 393216 tid 1 2024-04-18T21:51:45.942 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.942 INFO:tasks.rados.rados.0.smithi097.stdout:5: finishing write tid 3 to smithi09731216-5 2024-04-18T21:51:45.942 INFO:tasks.rados.rados.0.smithi097.stdout:5: oid 5 updating version 0 to 3 2024-04-18T21:51:45.942 INFO:tasks.rados.rados.0.smithi097.stdout:5: oid 5 updating version 3 to 4 2024-04-18T21:51:45.943 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 5 v 4 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-18T21:51:45.943 INFO:tasks.rados.rados.0.smithi097.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-18T21:51:45.945 INFO:tasks.rados.rados.0.smithi097.stdout:5: done (15 left) 2024-04-18T21:51:45.945 INFO:tasks.rados.rados.0.smithi097.stdout:25: write initial oid 25 2024-04-18T21:51:45.945 INFO:tasks.rados.rados.0.smithi097.stdout:25: seq_num 24 ranges {0=409600} 2024-04-18T21:51:45.946 INFO:tasks.rados.rados.0.smithi097.stdout:25: writing smithi09731216-25 from 0 to 409600 tid 1 2024-04-18T21:51:45.946 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:45.947 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:51:45.947 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:51:45.964 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:51:45.965 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:51:45.985 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:45.985 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:45.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:51:46.037 INFO:tasks.rados.rados.0.smithi097.stdout:4: finishing write tid 3 to smithi09731216-4 2024-04-18T21:51:46.037 INFO:tasks.rados.rados.0.smithi097.stdout:4: oid 4 updating version 0 to 1 2024-04-18T21:51:46.038 INFO:tasks.rados.rados.0.smithi097.stdout:4: oid 4 updating version 1 to 2 2024-04-18T21:51:46.038 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-18T21:51:46.038 INFO:tasks.rados.rados.0.smithi097.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-18T21:51:46.038 INFO:tasks.rados.rados.0.smithi097.stdout:4: done (15 left) 2024-04-18T21:51:46.038 INFO:tasks.rados.rados.0.smithi097.stdout:26: write initial oid 26 2024-04-18T21:51:46.038 INFO:tasks.rados.rados.0.smithi097.stdout:26: seq_num 25 ranges {0=425984} 2024-04-18T21:51:46.038 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:46.035+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:46.041 INFO:tasks.rados.rados.0.smithi097.stdout:26: writing smithi09731216-26 from 0 to 425984 tid 1 2024-04-18T21:51:46.041 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.041 INFO:tasks.rados.rados.0.smithi097.stdout:11: finishing write tid 3 to smithi09731216-11 2024-04-18T21:51:46.041 INFO:tasks.rados.rados.0.smithi097.stdout:11: oid 11 updating version 0 to 3 2024-04-18T21:51:46.041 INFO:tasks.rados.rados.0.smithi097.stdout:11: oid 11 updating version 3 to 4 2024-04-18T21:51:46.041 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 11 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-18T21:51:46.041 INFO:tasks.rados.rados.0.smithi097.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-18T21:51:46.041 INFO:tasks.rados.rados.0.smithi097.stdout:19: finishing write tid 1 to smithi09731216-19 2024-04-18T21:51:46.042 INFO:tasks.rados.rados.0.smithi097.stdout:11: done (15 left) 2024-04-18T21:51:46.042 INFO:tasks.rados.rados.0.smithi097.stdout:27: write initial oid 27 2024-04-18T21:51:46.042 INFO:tasks.rados.rados.0.smithi097.stdout:27: seq_num 26 ranges {0=442368} 2024-04-18T21:51:46.044 INFO:tasks.rados.rados.0.smithi097.stdout:27: writing smithi09731216-27 from 0 to 442368 tid 1 2024-04-18T21:51:46.044 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.045 INFO:tasks.rados.rados.0.smithi097.stdout:18: finishing write tid 3 to smithi09731216-18 2024-04-18T21:51:46.045 INFO:tasks.rados.rados.0.smithi097.stdout:18: oid 18 updating version 0 to 3 2024-04-18T21:51:46.045 INFO:tasks.rados.rados.0.smithi097.stdout:18: oid 18 updating version 3 to 4 2024-04-18T21:51:46.046 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 18 v 4 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-18T21:51:46.046 INFO:tasks.rados.rados.0.smithi097.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-18T21:51:46.046 INFO:tasks.rados.rados.0.smithi097.stdout:18: done (15 left) 2024-04-18T21:51:46.046 INFO:tasks.rados.rados.0.smithi097.stdout:28: write initial oid 28 2024-04-18T21:51:46.046 INFO:tasks.rados.rados.0.smithi097.stdout:28: seq_num 27 ranges {0=458752} 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:28: writing smithi09731216-28 from 0 to 458752 tid 1 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:7: finishing write tid 3 to smithi09731216-7 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:7: oid 7 updating version 0 to 4 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:7: oid 7 version 4 is already newer than 3 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 7 v 4 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:7: done (15 left) 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:29: write initial oid 29 2024-04-18T21:51:46.050 INFO:tasks.rados.rados.0.smithi097.stdout:29: seq_num 28 ranges {0=475136} 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:29: writing smithi09731216-29 from 0 to 475136 tid 1 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:12: finishing write tid 3 to smithi09731216-12 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:12: oid 12 updating version 0 to 3 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:12: oid 12 updating version 3 to 4 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 12 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:12: done (15 left) 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:30: write initial oid 30 2024-04-18T21:51:46.054 INFO:tasks.rados.rados.0.smithi097.stdout:30: seq_num 29 ranges {0=491520} 2024-04-18T21:51:46.057 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:51:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:51:46.057 INFO:tasks.rados.rados.0.smithi097.stdout:30: writing smithi09731216-30 from 0 to 491520 tid 1 2024-04-18T21:51:46.057 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.124 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:46.124+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:46.135 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:46.136 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:46.141 INFO:tasks.rados.rados.0.smithi097.stdout:19: finishing write tid 2 to smithi09731216-19 2024-04-18T21:51:46.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:51:46.154 INFO:tasks.rados.rados.0.smithi097.stdout:3: finishing write tid 2 to smithi09731216-3 2024-04-18T21:51:46.154 INFO:tasks.rados.rados.0.smithi097.stdout:6: finishing write tid 1 to smithi09731216-6 2024-04-18T21:51:46.155 INFO:tasks.rados.rados.0.smithi097.stdout:6: finishing write tid 2 to smithi09731216-6 2024-04-18T21:51:46.156 INFO:tasks.rados.rados.0.smithi097.stdout:27: finishing write tid 1 to smithi09731216-27 2024-04-18T21:51:46.162 INFO:tasks.rados.rados.0.smithi097.stdout:15: finishing write tid 1 to smithi09731216-15 2024-04-18T21:51:46.163 INFO:tasks.rados.rados.0.smithi097.stdout:15: finishing write tid 2 to smithi09731216-15 2024-04-18T21:51:46.179 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:51:46.179 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:51:46.203 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:51:46.203 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:51:46.225 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:46.223+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:46.232 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:51:46.232 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:51:46.238 INFO:tasks.rados.rados.0.smithi097.stdout:28: finishing write tid 1 to smithi09731216-28 2024-04-18T21:51:46.241 INFO:tasks.rados.rados.0.smithi097.stdout:21: finishing write tid 1 to smithi09731216-21 2024-04-18T21:51:46.242 INFO:tasks.rados.rados.0.smithi097.stdout:21: finishing write tid 2 to smithi09731216-21 2024-04-18T21:51:46.245 INFO:tasks.rados.rados.0.smithi097.stdout:19: finishing write tid 3 to smithi09731216-19 2024-04-18T21:51:46.245 INFO:tasks.rados.rados.0.smithi097.stdout:19: oid 19 updating version 0 to 1 2024-04-18T21:51:46.245 INFO:tasks.rados.rados.0.smithi097.stdout:19: oid 19 updating version 1 to 2 2024-04-18T21:51:46.246 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 19 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-18T21:51:46.246 INFO:tasks.rados.rados.0.smithi097.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-18T21:51:46.246 INFO:tasks.rados.rados.0.smithi097.stdout:19: done (15 left) 2024-04-18T21:51:46.246 INFO:tasks.rados.rados.0.smithi097.stdout:31: write initial oid 31 2024-04-18T21:51:46.246 INFO:tasks.rados.rados.0.smithi097.stdout:31: seq_num 30 ranges {0=507904} 2024-04-18T21:51:46.253 INFO:tasks.rados.rados.0.smithi097.stdout:31: writing smithi09731216-31 from 0 to 507904 tid 1 2024-04-18T21:51:46.253 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.263 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:51:46.263 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:51:46.266 INFO:tasks.rados.rados.0.smithi097.stdout:17: finishing write tid 1 to smithi09731216-17 2024-04-18T21:51:46.268 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:46.268 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:46.268 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:46.268 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:46.268 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:46.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:51:46.290 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:51:46.290 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:51:46.317 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:51:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:51:46.326 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:46.323+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:46.338 INFO:tasks.rados.rados.0.smithi097.stdout:28: finishing write tid 2 to smithi09731216-28 2024-04-18T21:51:46.346 INFO:tasks.rados.rados.0.smithi097.stdout:23: finishing write tid 1 to smithi09731216-23 2024-04-18T21:51:46.346 INFO:tasks.rados.rados.0.smithi097.stdout:23: finishing write tid 2 to smithi09731216-23 2024-04-18T21:51:46.346 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:51:46.347 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:51:46.347 INFO:tasks.rados.rados.0.smithi097.stdout:26: finishing write tid 1 to smithi09731216-26 2024-04-18T21:51:46.356 INFO:tasks.rados.rados.0.smithi097.stdout:27: finishing write tid 2 to smithi09731216-27 2024-04-18T21:51:46.366 INFO:tasks.rados.rados.0.smithi097.stdout:17: finishing write tid 2 to smithi09731216-17 2024-04-18T21:51:46.374 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:51:46.374 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:51:46.437 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:46.435+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:46.445 INFO:tasks.rados.rados.0.smithi097.stdout:30: finishing write tid 1 to smithi09731216-30 2024-04-18T21:51:46.446 INFO:tasks.rados.rados.0.smithi097.stdout:30: finishing write tid 2 to smithi09731216-30 2024-04-18T21:51:46.447 INFO:tasks.rados.rados.0.smithi097.stdout:26: finishing write tid 2 to smithi09731216-26 2024-04-18T21:51:46.449 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:46.449 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es3 30/26 ECSubWrite(tid=9, reqid=client.4509.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.339583+0000", 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.110727741, 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000958931, 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.339583+0000", 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.339583+0000", 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.339585+0000", 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.801e-06 2024-04-18T21:51:46.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.339606+0000", 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1331e-05 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.339627+0000", 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0723000000000001e-05 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.339639+0000", 2024-04-18T21:51:46.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2629000000000001e-05 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.339760+0000", 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121049 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.339778+0000", 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7557000000000001e-05 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.340486+0000", 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00070861700000000004 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.340542+0000", 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5223999999999998e-05 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s6 30/26 ECSubWrite(tid=9, reqid=client.4509.0:68, at_version=0'0, trim_to=0'0, roll_forward_to=29'6))", 2024-04-18T21:51:46.454 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.347874+0000", 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.102436112, 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035273470000000001, 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.347874+0000", 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.347874+0000", 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.347875+0000", 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0130000000000001e-06 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.347889+0000", 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3888e-05 2024-04-18T21:51:46.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.347908+0000", 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8440999999999998e-05 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.347921+0000", 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2979999999999999e-05 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348003+0000", 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2157999999999995e-05 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348017+0000", 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4622e-05 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.351343+0000", 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003325125 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.351402+0000", 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9120000000000003e-05 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.459 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:46.464 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:51:46.467 INFO:tasks.rados.rados.0.smithi097.stdout:20: finishing write tid 1 to smithi09731216-20 2024-04-18T21:51:46.528 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:46.523+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:46.545 INFO:tasks.rados.rados.0.smithi097.stdout:21: finishing write tid 3 to smithi09731216-21 2024-04-18T21:51:46.545 INFO:tasks.rados.rados.0.smithi097.stdout:21: oid 21 updating version 0 to 4 2024-04-18T21:51:46.546 INFO:tasks.rados.rados.0.smithi097.stdout:21: oid 21 version 4 is already newer than 3 2024-04-18T21:51:46.546 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 21 v 4 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-18T21:51:46.546 INFO:tasks.rados.rados.0.smithi097.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-18T21:51:46.546 INFO:tasks.rados.rados.0.smithi097.stdout:21: done (15 left) 2024-04-18T21:51:46.546 INFO:tasks.rados.rados.0.smithi097.stdout:32: write initial oid 32 2024-04-18T21:51:46.546 INFO:tasks.rados.rados.0.smithi097.stdout:32: seq_num 31 ranges {0=524288} 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:32: writing smithi09731216-32 from 0 to 524288 tid 1 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:23: finishing write tid 3 to smithi09731216-23 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:23: oid 23 updating version 0 to 6 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:23: oid 23 version 6 is already newer than 5 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:23: done (15 left) 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:33: write initial oid 33 2024-04-18T21:51:46.555 INFO:tasks.rados.rados.0.smithi097.stdout:33: seq_num 32 ranges {0=540672} 2024-04-18T21:51:46.564 INFO:tasks.rados.rados.0.smithi097.stdout:33: writing smithi09731216-33 from 0 to 540672 tid 1 2024-04-18T21:51:46.564 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.588 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:46.588 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:46.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.588 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.5s0 30/26 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.559363+0000", 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.029827453, 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.029849823000000001, 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559363+0000", 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559363+0000", 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559364+0000", 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.093e-06 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559384+0000", 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9959999999999999e-05 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559411+0000", 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6472999999999999e-05 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559426+0000", 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5506999999999999e-05 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.591 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=16, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.559400+0000", 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.029790919999999999, 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.029853278, 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "no flag points reached", 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559400+0000", 2024-04-18T21:51:46.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559400+0000", 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559401+0000", 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.083e-06 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559407+0000", 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2670000000000003e-06 2024-04-18T21:51:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559420+0000", 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3298e-05 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 30/26 ECSubWriteReply(tid=18, last_complete=30'4, committed=1, applied=1))", 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.559460+0000", 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.029730177999999999, 2024-04-18T21:51:46.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.029822187, 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "no flag points reached", 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559460+0000", 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559460+0000", 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559461+0000", 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1399999999999996e-07 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559468+0000", 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6660000000000002e-06 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559486+0000", 2024-04-18T21:51:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7654000000000001e-05 2024-04-18T21:51:46.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.597 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.597 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:51:46.597 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 3 2024-04-18T21:51:46.597 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:46.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:51:46.629 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:46.627+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:46.642 INFO:tasks.rados.rados.0.smithi097.stdout:28: finishing write tid 3 to smithi09731216-28 2024-04-18T21:51:46.642 INFO:tasks.rados.rados.0.smithi097.stdout:28: oid 28 updating version 0 to 5 2024-04-18T21:51:46.642 INFO:tasks.rados.rados.0.smithi097.stdout:28: oid 28 updating version 5 to 6 2024-04-18T21:51:46.642 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-18T21:51:46.642 INFO:tasks.rados.rados.0.smithi097.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-18T21:51:46.642 INFO:tasks.rados.rados.0.smithi097.stdout:28: done (15 left) 2024-04-18T21:51:46.642 INFO:tasks.rados.rados.0.smithi097.stdout:34: write initial oid 34 2024-04-18T21:51:46.642 INFO:tasks.rados.rados.0.smithi097.stdout:34: seq_num 33 ranges {0=557056} 2024-04-18T21:51:46.651 INFO:tasks.rados.rados.0.smithi097.stdout:34: writing smithi09731216-34 from 0 to 557056 tid 1 2024-04-18T21:51:46.651 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.651 INFO:tasks.rados.rados.0.smithi097.stdout:30: finishing write tid 3 to smithi09731216-30 2024-04-18T21:51:46.651 INFO:tasks.rados.rados.0.smithi097.stdout:30: oid 30 updating version 0 to 3 2024-04-18T21:51:46.652 INFO:tasks.rados.rados.0.smithi097.stdout:30: oid 30 updating version 3 to 4 2024-04-18T21:51:46.652 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 30 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-18T21:51:46.652 INFO:tasks.rados.rados.0.smithi097.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-18T21:51:46.652 INFO:tasks.rados.rados.0.smithi097.stdout:30: done (15 left) 2024-04-18T21:51:46.652 INFO:tasks.rados.rados.0.smithi097.stdout:35: write initial oid 35 2024-04-18T21:51:46.652 INFO:tasks.rados.rados.0.smithi097.stdout:35: seq_num 34 ranges {0=573440} 2024-04-18T21:51:46.662 INFO:tasks.rados.rados.0.smithi097.stdout:35: writing smithi09731216-35 from 0 to 573440 tid 1 2024-04-18T21:51:46.662 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.662 INFO:tasks.rados.rados.0.smithi097.stdout:27: finishing write tid 3 to smithi09731216-27 2024-04-18T21:51:46.662 INFO:tasks.rados.rados.0.smithi097.stdout:27: oid 27 updating version 0 to 6 2024-04-18T21:51:46.662 INFO:tasks.rados.rados.0.smithi097.stdout:27: oid 27 version 6 is already newer than 5 2024-04-18T21:51:46.662 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-18T21:51:46.663 INFO:tasks.rados.rados.0.smithi097.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-18T21:51:46.663 INFO:tasks.rados.rados.0.smithi097.stdout:27: done (15 left) 2024-04-18T21:51:46.663 INFO:tasks.rados.rados.0.smithi097.stdout:36: write initial oid 36 2024-04-18T21:51:46.663 INFO:tasks.rados.rados.0.smithi097.stdout:36: seq_num 35 ranges {0=589824} 2024-04-18T21:51:46.673 INFO:tasks.rados.rados.0.smithi097.stdout:36: writing smithi09731216-36 from 0 to 589824 tid 1 2024-04-18T21:51:46.674 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.674 INFO:tasks.rados.rados.0.smithi097.stdout:24: finishing write tid 1 to smithi09731216-24 2024-04-18T21:51:46.674 INFO:tasks.rados.rados.0.smithi097.stdout:24: finishing write tid 2 to smithi09731216-24 2024-04-18T21:51:46.726 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:46.726 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:46.727 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:46.727 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:46.727 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:46.730 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:46.727+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:46.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:51:46.831 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:46.832+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:46.863 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:46.863 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:46.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:46.863 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:46.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.863 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 30/26 ECSubWriteReply(tid=15, last_complete=30'4, committed=1, applied=1))", 2024-04-18T21:51:46.863 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.248977+0000", 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.61179462799999995, 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099031392999999995, 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.248977+0000", 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.248977+0000", 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.248980+0000", 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2500000000000001e-06 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.248992+0000", 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1936000000000001e-05 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.249021+0000", 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9533e-05 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.249037+0000", 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5577e-05 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.347982+0000", 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098945416999999994 2024-04-18T21:51:46.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348009+0000", 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6679999999999999e-05 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=14, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.249372+0000", 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.61139992799999998, 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.108641424, 2024-04-18T21:51:46.867 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.249372+0000", 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.249372+0000", 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.249374+0000", 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5570000000000001e-06 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.249382+0000", 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2730000000000002e-06 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.249406+0000", 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4383e-05 2024-04-18T21:51:46.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.249420+0000", 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3667e-05 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357331+0000", 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107911294 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358014+0000", 2024-04-18T21:51:46.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068225000000000002 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 30/26 ECSubWriteReply(tid=15, last_complete=30'4, committed=1, applied=1))", 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.257659+0000", 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.60311293600000004, 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.090557444000000001, 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.871 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257659+0000", 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257659+0000", 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257660+0000", 2024-04-18T21:51:46.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1200000000000002e-07 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257665+0000", 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0080000000000002e-06 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257682+0000", 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7512000000000001e-05 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257690+0000", 2024-04-18T21:51:46.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8590000000000005e-06 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348191+0000", 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.090500782000000002 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348217+0000", 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5571e-05 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 30/26 ECSubWriteReply(tid=15, last_complete=30'4, committed=1, applied=1))", 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.257718+0000", 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.60305444600000002, 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19131084000000001, 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257718+0000", 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257718+0000", 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257718+0000", 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3700000000000001e-07 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257721+0000", 2024-04-18T21:51:46.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4260000000000002e-06 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257728+0000", 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5839999999999997e-06 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257734+0000", 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4210000000000003e-06 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448334+0000", 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.190600884 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449029+0000", 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069408800000000004 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s7 30/26 ECSubWrite(tid=5, reqid=client.4509.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:46.878 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.257748+0000", 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.60302423800000005, 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011948029999999999, 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257748+0000", 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257748+0000", 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257748+0000", 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5699999999999998e-07 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257760+0000", 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1895e-05 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257772+0000", 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1491999999999999e-05 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257780+0000", 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7420000000000003e-06 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257902+0000", 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121721 2024-04-18T21:51:46.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.257920+0000", 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7963000000000001e-05 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.258853+0000", 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093320999999999996 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.258943+0000", 2024-04-18T21:51:46.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9423000000000001e-05 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s5 29/26 ECSubRead(tid=11, to_read={3:c022028e:::smithi09731216-15:head=0,4096,0}, subchunks={3:c022028e:::smithi09731216-15:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.264919+0000", 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.59585280100000004, 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060754099999999996, 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.883 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264919+0000", 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264919+0000", 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264921+0000", 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.686e-06 2024-04-18T21:51:46.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264940+0000", 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8790000000000001e-05 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264969+0000", 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9603000000000001e-05 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264990+0000", 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0302999999999999e-05 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265132+0000", 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014246199999999999 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265527+0000", 2024-04-18T21:51:46.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00039469700000000001 2024-04-18T21:51:46.887 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.889 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.889 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.889 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 30/26 ECSubWrite(tid=9, reqid=client.4509.0:68, at_version=0'0, trim_to=0'0, roll_forward_to=29'6))", 2024-04-18T21:51:46.889 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.348003+0000", 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.51276962100000001, 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00080316100000000004, 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348003+0000", 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348003+0000", 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348004+0000", 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1689999999999999e-06 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348020+0000", 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6203000000000001e-05 2024-04-18T21:51:46.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348041+0000", 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1101999999999999e-05 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348056+0000", 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5336999999999999e-05 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348161+0000", 2024-04-18T21:51:46.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104217 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348175+0000", 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4185000000000001e-05 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348754+0000", 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057879999999999997 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348806+0000", 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2148000000000003e-05 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.cs0 30/26 ECSubRead(tid=17, to_read={3:3bc99ad3:::smithi09731216-27:head=0,4096,0}, subchunks={3:3bc99ad3:::smithi09731216-27:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.358324+0000", 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.50244795099999995, 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040643, 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.894 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358324+0000", 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358324+0000", 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358324+0000", 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358324+0000", 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358355+0000", 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1090999999999998e-05 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358373+0000", 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7747000000000001e-05 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358448+0000", 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4776000000000004e-05 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.358731+0000", 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028281600000000002 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.897 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=16, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.359693+0000", 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.501079308, 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10020987200000001, 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.359693+0000", 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.359693+0000", 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.359694+0000", 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.026e-06 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.359698+0000", 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4479999999999996e-06 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.359710+0000", 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1379999999999999e-05 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.359717+0000", 2024-04-18T21:51:46.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0029999999999997e-06 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459870+0000", 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10015315499999999 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459903+0000", 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.286e-05 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s5 29/26 ECSubRead(tid=12, to_read={3:c057a400:::smithi09731216-17:head=0,4096,0}, subchunks={3:c057a400:::smithi09731216-17:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.367965+0000", 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.49280692700000001, 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045449999999999999, 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367965+0000", 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367965+0000", 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367967+0000", 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7460000000000001e-06 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367984+0000", 2024-04-18T21:51:46.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7173999999999999e-05 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368017+0000", 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2820000000000001e-05 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368035+0000", 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7886000000000001e-05 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368135+0000", 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010050199999999999 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368420+0000", 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028437200000000002 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=16, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.445979+0000", 2024-04-18T21:51:46.905 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.41479347599999999, 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014050875000000001, 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.445979+0000", 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.445979+0000", 2024-04-18T21:51:46.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.445980+0000", 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7600000000000001e-06 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.445991+0000", 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0383e-05 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.446022+0000", 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1517000000000003e-05 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.446040+0000", 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7317e-05 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459990+0000", 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013950842 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.460030+0000", 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9056e-05 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.fs5 30/26 ECSubWrite(tid=143, reqid=client.4509.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=30'4))", 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.447499+0000", 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.41327361099999999, 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001227077, 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.909 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.447499+0000", 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.447499+0000", 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.447500+0000", 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.657e-06 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.447527+0000", 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6746000000000001e-05 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.447558+0000", 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0620000000000002e-05 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.447577+0000", 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9187000000000001e-05 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.447669+0000", 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2091999999999999e-05 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.447679+0000", 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0118999999999999e-05 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448647+0000", 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00096827999999999997 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448726+0000", 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8375999999999996e-05 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.fs5 30/26 ECSubRead(tid=144, to_read={3:f240687b:::smithi09731216-30:head=0,4096,0}, subchunks={3:f240687b:::smithi09731216-30:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:46.913 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.448659+0000", 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.41211301299999997, 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00078032399999999999, 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448659+0000", 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448659+0000", 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448661+0000", 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.57e-06 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448680+0000", 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9249000000000002e-05 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448722+0000", 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2317000000000002e-05 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448748+0000", 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6227000000000001e-05 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.448842+0000", 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3148999999999999e-05 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449439+0000", 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00059781199999999997 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.9s4 30/26 ECSubRead(tid=10, to_read={3:9da3af1e:::smithi09731216-23:head=0,4096,0}, subchunks={3:9da3af1e:::smithi09731216-23:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.449352+0000", 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.41141965899999999, 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021220039999999998, 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449352+0000", 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449352+0000", 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449354+0000", 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7039999999999999e-06 2024-04-18T21:51:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449371+0000", 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6786000000000002e-05 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449412+0000", 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0673000000000002e-05 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449440+0000", 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8266999999999999e-05 2024-04-18T21:51:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449533+0000", 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2663000000000004e-05 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.451474+0000", 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019419109999999999 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.5s0 30/26 ECSubRead(tid=19, to_read={3:ab33d1b5:::smithi09731216-26:head=0,4096,0}, subchunks={3:ab33d1b5:::smithi09731216-26:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.449750+0000", 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.411022002, 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00083908400000000003, 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449750+0000", 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449750+0000", 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449750+0000", 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449750+0000", 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449924+0000", 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017385600000000001 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449950+0000", 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5833000000000002e-05 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450125+0000", 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000175355 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450589+0000", 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046403999999999998 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 30/26 ECSubWriteReply(tid=18, last_complete=30'4, committed=1, applied=1))", 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.450457+0000", 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.41031557499999999, 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000204976, 2024-04-18T21:51:46.924 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450457+0000", 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450457+0000", 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450457+0000", 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3099999999999997e-07 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450461+0000", 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.439e-06 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450470+0000", 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1789999999999997e-06 2024-04-18T21:51:46.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450475+0000", 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.48e-06 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450633+0000", 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015754399999999999 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450662+0000", 2024-04-18T21:51:46.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8603e-05 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.5s0 30/26 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.450566+0000", 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.41020595399999998, 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026325099999999999, 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450566+0000", 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450566+0000", 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450566+0000", 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450566+0000", 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450608+0000", 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2225999999999999e-05 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450625+0000", 2024-04-18T21:51:46.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7062999999999999e-05 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450735+0000", 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110009 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450829+0000", 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3953e-05 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.cs0 30/26 ECSubReadReply(tid=17, attrs_read=0))", 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.459102+0000", 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.40167005, 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010385850000000001, 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459102+0000", 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459102+0000", 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459103+0000", 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2219999999999999e-06 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459122+0000", 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8485e-05 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459141+0000", 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9037000000000001e-05 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459155+0000", 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4054e-05 2024-04-18T21:51:46.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.460076+0000", 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092074099999999999 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.460141+0000", 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5046e-05 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=16, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.459175+0000", 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.40159705499999998, 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100251692, 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459175+0000", 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459175+0000", 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459175+0000", 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2500000000000001e-07 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459179+0000", 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9199999999999997e-06 2024-04-18T21:51:46.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459187+0000", 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8059999999999995e-06 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559313+0000", 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100126007 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559412+0000", 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8633000000000002e-05 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.559427+0000", 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4901e-05 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s5 30/26 ECSubWrite(tid=13, reqid=client.4509.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=29'9))", 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.469054+0000", 2024-04-18T21:51:46.939 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.39171779200000001, 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00085393000000000003, 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469054+0000", 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469054+0000", 2024-04-18T21:51:46.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469055+0000", 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.079e-06 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469071+0000", 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5903999999999999e-05 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469096+0000", 2024-04-18T21:51:46.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4224999999999999e-05 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469107+0000", 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1698999999999999e-05 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469204+0000", 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6656000000000004e-05 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469215+0000", 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0729e-05 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469867+0000", 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000652529 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469908+0000", 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1109000000000002e-05 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.943 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.944 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.944 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:46.944 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:46.944 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:46.946 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:51:46.946 INFO:tasks.rados.rados.0.smithi097.stdout:3: finishing write tid 3 to smithi09731216-3 2024-04-18T21:51:46.946 INFO:tasks.rados.rados.0.smithi097.stdout:3: oid 3 updating version 0 to 2 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:3: oid 3 version 2 is already newer than 1 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:6: finishing write tid 3 to smithi09731216-6 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:6: oid 6 updating version 0 to 3 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:6: oid 6 updating version 3 to 4 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 6 v 4 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:3: done (15 left) 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:6: done (14 left) 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:37: write initial oid 37 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:37: seq_num 36 ranges {0=606208} 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:37: writing smithi09731216-37 from 0 to 606208 tid 1 2024-04-18T21:51:46.947 INFO:tasks.rados.rados.0.smithi097.stdout:38: write initial oid 38 2024-04-18T21:51:46.948 INFO:tasks.rados.rados.0.smithi097.stdout:38: seq_num 37 ranges {0=622592} 2024-04-18T21:51:46.948 INFO:tasks.rados.rados.0.smithi097.stdout:38: writing smithi09731216-38 from 0 to 622592 tid 1 2024-04-18T21:51:46.948 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:46.948 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:46.948+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:46.966 INFO:tasks.rados.rados.0.smithi097.stdout:32: finishing write tid 1 to smithi09731216-32 2024-04-18T21:51:46.966 INFO:tasks.rados.rados.0.smithi097.stdout:32: finishing write tid 2 to smithi09731216-32 2024-04-18T21:51:47.046 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:47.048+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:47.080 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:51:47.080 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:47.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:51:47.146 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:47.143+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:47.165 INFO:tasks.rados.rados.0.smithi097.stdout:34: finishing write tid 1 to smithi09731216-34 2024-04-18T21:51:47.169 INFO:tasks.rados.rados.0.smithi097.stdout:24: finishing write tid 3 to smithi09731216-24 2024-04-18T21:51:47.169 INFO:tasks.rados.rados.0.smithi097.stdout:24: oid 24 updating version 0 to 4 2024-04-18T21:51:47.170 INFO:tasks.rados.rados.0.smithi097.stdout:24: oid 24 version 4 is already newer than 3 2024-04-18T21:51:47.170 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-18T21:51:47.170 INFO:tasks.rados.rados.0.smithi097.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-18T21:51:47.170 INFO:tasks.rados.rados.0.smithi097.stdout:24: done (15 left) 2024-04-18T21:51:47.170 INFO:tasks.rados.rados.0.smithi097.stdout:39: write initial oid 39 2024-04-18T21:51:47.170 INFO:tasks.rados.rados.0.smithi097.stdout:39: seq_num 38 ranges {0=638976} 2024-04-18T21:51:47.180 INFO:tasks.rados.rados.0.smithi097.stdout:39: writing smithi09731216-39 from 0 to 638976 tid 1 2024-04-18T21:51:47.180 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:47.217 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:47.217 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:47.218 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:47.218 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:47.218 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:47.231 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:51:47.247 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:47.243+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:47.269 INFO:tasks.rados.rados.0.smithi097.stdout:34: finishing write tid 2 to smithi09731216-34 2024-04-18T21:51:47.348 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:47.348+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:47.397 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 30/26 ECSubWrite(tid=5, reqid=client.4509.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.243686+0000", 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.1526297750000001, 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001321767, 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.398 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.243686+0000", 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.243686+0000", 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.243688+0000", 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.863e-06 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.243715+0000", 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6440000000000001e-05 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.243746+0000", 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1457000000000001e-05 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.243773+0000", 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6440000000000001e-05 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.243913+0000", 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014081799999999999 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.243929+0000", 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5987000000000001e-05 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.244966+0000", 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010364930000000001 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.245008+0000", 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2268999999999998e-05 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s7 30/26 ECSubWrite(tid=9, reqid=client.4509.0:68, at_version=0'0, trim_to=0'0, roll_forward_to=29'6))", 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.348145+0000", 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.048171586, 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00080306799999999999, 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348145+0000", 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348145+0000", 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348146+0000", 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.494e-06 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348164+0000", 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7997000000000001e-05 2024-04-18T21:51:47.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348188+0000", 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3519999999999998e-05 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348203+0000", 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5162e-05 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348302+0000", 2024-04-18T21:51:47.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8720000000000003e-05 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348313+0000", 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1046999999999999e-05 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348890+0000", 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057710400000000005 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.348948+0000", 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8023999999999999e-05 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.cs2 30/26 ECSubWrite(tid=16, reqid=client.4509.0:80, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.357836+0000", 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.0384798289999999, 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101219042, 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.407 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357836+0000", 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357836+0000", 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357837+0000", 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.066e-06 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357853+0000", 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5255e-05 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357874+0000", 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.137e-05 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357885+0000", 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0604e-05 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.458089+0000", 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100204521 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.458105+0000", 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5957e-05 2024-04-18T21:51:47.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459001+0000", 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089603999999999997 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459055+0000", 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4228999999999999e-05 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s3 30/26 ECSubWrite(tid=18, reqid=client.4509.0:77, at_version=0'0, trim_to=0'0, roll_forward_to=30'4))", 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.449034+0000", 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.94728242600000001, 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001350752, 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449034+0000", 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449034+0000", 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449036+0000", 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.666e-06 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449067+0000", 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0573e-05 2024-04-18T21:51:47.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449118+0000", 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1122999999999998e-05 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449145+0000", 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.688e-05 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449293+0000", 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000147789 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.449334+0000", 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1502999999999999e-05 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450313+0000", 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00097819899999999991 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.450385+0000", 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2019000000000003e-05 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s3 30/26 ECSubWrite(tid=13, reqid=client.4509.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=29'9))", 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.469106+0000", 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.92721033600000002, 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012107629999999999, 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.416 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469106+0000", 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469106+0000", 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469107+0000", 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5549999999999999e-06 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469126+0000", 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8643000000000001e-05 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469156+0000", 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9417e-05 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469170+0000", 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.465e-05 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469263+0000", 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3228999999999998e-05 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469277+0000", 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3550000000000001e-05 2024-04-18T21:51:47.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.470266+0000", 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0009889670000000001 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.470317+0000", 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0751999999999998e-05 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 30/26 ECSubWrite(tid=7, reqid=client.4509.0:94, at_version=30'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.560058+0000", 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.83625850599999996, 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001187465, 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.560058+0000", 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.560058+0000", 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.560059+0000", 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.22e-06 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.560298+0000", 2024-04-18T21:51:47.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023922200000000001 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.560315+0000", 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6982000000000001e-05 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.560326+0000", 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0794e-05 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.560393+0000", 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6688999999999998e-05 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.560401+0000", 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0260000000000007e-06 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.561195+0000", 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00079456199999999996 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.561245+0000", 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9969999999999998e-05 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.562752+0000", 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.83356448699999997, 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201174983, 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.425 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562752+0000", 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562752+0000", 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562752+0000", 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5199999999999997e-07 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562767+0000", 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4399999999999999e-05 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562778+0000", 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1225e-05 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562784+0000", 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5060000000000001e-06 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763008+0000", 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200223922 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763022+0000", 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3551e-05 2024-04-18T21:51:47.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763889+0000", 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00086700299999999998 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763927+0000", 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7824000000000002e-05 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 30/26 ECSubWrite(tid=11, reqid=client.4509.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.668797+0000", 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.72751895200000005, 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10074586200000001, 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668797+0000", 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668797+0000", 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668799+0000", 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9269999999999999e-06 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668817+0000", 2024-04-18T21:51:47.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8213e-05 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668839+0000", 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1661000000000001e-05 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.768915+0000", 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100075514 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.768979+0000", 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4796999999999996e-05 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.768988+0000", 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9169999999999995e-06 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.769509+0000", 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052097899999999997 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.769543+0000", 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3853999999999998e-05 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.cs2 30/26 ECSubWrite(tid=20, reqid=client.4509.0:100, at_version=30'7, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.762156+0000", 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.63415988099999998, 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001314391, 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.434 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762156+0000", 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762156+0000", 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762158+0000", 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.637e-06 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762267+0000", 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108726 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762287+0000", 2024-04-18T21:51:47.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0682e-05 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762301+0000", 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4079e-05 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762428+0000", 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012620900000000001 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762444+0000", 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6256000000000002e-05 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763424+0000", 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00098031500000000005 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763471+0000", 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6486999999999997e-05 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.438 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.cs2 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.763531+0000", 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.63278566199999997, 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001019282, 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763531+0000", 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763531+0000", 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763531+0000", 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0599999999999999e-07 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3263000000000001e-05 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763574+0000", 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9468999999999999e-05 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763587+0000", 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2615e-05 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763738+0000", 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000151111 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763749+0000", 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0733999999999999e-05 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.764496+0000", 2024-04-18T21:51:47.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074703300000000003 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.764550+0000", 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4150999999999997e-05 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 30/26 ECSubWrite(tid=7, reqid=client.4509.0:106, at_version=30'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.768996+0000", 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.62732017600000001, 2024-04-18T21:51:47.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013278840000000001, 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.768996+0000", 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.768996+0000", 2024-04-18T21:51:47.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.768996+0000", 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.04e-07 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.769162+0000", 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016619599999999999 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.769173+0000", 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0793e-05 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.769180+0000", 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1729999999999998e-06 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.769259+0000", 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8075e-05 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.769272+0000", 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3127e-05 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.770284+0000", 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00101235 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.770324+0000", 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9966000000000003e-05 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.447 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 30/26 ECSubWrite(tid=8, reqid=client.4509.0:107, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.783191+0000", 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.613124899, 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10112057000000001, 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.783191+0000", 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.783191+0000", 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.783193+0000", 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.251e-06 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.783216+0000", 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3699000000000001e-05 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.783238+0000", 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1784e-05 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.883411+0000", 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100172517 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.883513+0000", 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102748 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.883528+0000", 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4428e-05 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.884237+0000", 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00070945000000000003 2024-04-18T21:51:47.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.884312+0000", 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4692999999999998e-05 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 30/26 ECSubWrite(tid=9, reqid=client.4509.0:95, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.966703+0000", 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.429612775, 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103715603, 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.452 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966703+0000", 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966703+0000", 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966704+0000", 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9399999999999993e-07 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966726+0000", 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1616000000000001e-05 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966745+0000", 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.863e-05 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966763+0000", 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7816e-05 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.066982+0000", 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10021948 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.066999+0000", 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7479999999999999e-05 2024-04-18T21:51:47.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.070419+0000", 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003419587 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.456 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:47.457 INFO:tasks.rados.rados.0.smithi097.stdout:39: finishing write tid 1 to smithi09731216-39 2024-04-18T21:51:47.457 INFO:tasks.rados.rados.0.smithi097.stdout:39: finishing write tid 2 to smithi09731216-39 2024-04-18T21:51:47.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:51:47.459 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:47.460+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:47.558 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:47.555+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:47.584 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:47.584 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:47.584 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:51:47.584 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:47.587 INFO:tasks.rados.rados.0.smithi097.stdout:34: finishing write tid 3 to smithi09731216-34 2024-04-18T21:51:47.588 INFO:tasks.rados.rados.0.smithi097.stdout:34: oid 34 updating version 0 to 7 2024-04-18T21:51:47.588 INFO:tasks.rados.rados.0.smithi097.stdout:34: oid 34 updating version 7 to 8 2024-04-18T21:51:47.588 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 34 v 8 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-18T21:51:47.588 INFO:tasks.rados.rados.0.smithi097.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-18T21:51:47.588 INFO:tasks.rados.rados.0.smithi097.stdout:34: done (15 left) 2024-04-18T21:51:47.588 INFO:tasks.rados.rados.0.smithi097.stdout:40: write initial oid 40 2024-04-18T21:51:47.588 INFO:tasks.rados.rados.0.smithi097.stdout:40: seq_num 39 ranges {0=671744} 2024-04-18T21:51:47.598 INFO:tasks.rados.rados.0.smithi097.stdout:40: writing smithi09731216-40 from 0 to 671744 tid 1 2024-04-18T21:51:47.599 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:47.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:51:47.660 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:47.660+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:47.706 INFO:tasks.rados.rados.0.smithi097.stdout:39: finishing write tid 3 to smithi09731216-39 2024-04-18T21:51:47.706 INFO:tasks.rados.rados.0.smithi097.stdout:39: oid 39 updating version 0 to 6 2024-04-18T21:51:47.706 INFO:tasks.rados.rados.0.smithi097.stdout:39: oid 39 version 6 is already newer than 5 2024-04-18T21:51:47.706 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-18T21:51:47.706 INFO:tasks.rados.rados.0.smithi097.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-18T21:51:47.706 INFO:tasks.rados.rados.0.smithi097.stdout:39: done (15 left) 2024-04-18T21:51:47.706 INFO:tasks.rados.rados.0.smithi097.stdout:41: write initial oid 41 2024-04-18T21:51:47.706 INFO:tasks.rados.rados.0.smithi097.stdout:41: seq_num 40 ranges {0=688128} 2024-04-18T21:51:47.716 INFO:tasks.rados.rados.0.smithi097.stdout:41: writing smithi09731216-41 from 0 to 688128 tid 1 2024-04-18T21:51:47.716 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:47.719 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:47.719 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:47.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:51:47.780 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:47.779+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.054151035, 2024-04-18T21:51:47.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58281355499999998, 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 1 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798268+0000", 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5959999999999998e-06 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798283+0000", 2024-04-18T21:51:47.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5495999999999999e-05 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798322+0000", 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8899999999999997e-05 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798347+0000", 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5510000000000001e-05 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886734+0000", 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088386079000000006 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886749+0000", 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5019e-05 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.078589+0000", 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19184037900000001 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079052+0000", 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046354399999999998 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079551+0000", 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049870299999999997 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.083572+0000", 2024-04-18T21:51:47.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040212609999999999 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.380910+0000", 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29733748900000001 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381009+0000", 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9025999999999997e-05 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381079+0000", 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9553000000000003e-05 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.117341701000001, 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40660126499999999, 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.863 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 33 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:47.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735075+0000", 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4300000000000001e-07 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735081+0000", 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8669999999999999e-06 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735094+0000", 2024-04-18T21:51:47.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2813000000000001e-05 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735102+0000", 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4770000000000003e-06 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.737810+0000", 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027080089999999999 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.737953+0000", 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014291900000000001 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.738162+0000", 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020895500000000001 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.940717+0000", 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20255461299999999 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141518+0000", 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20080082699999999 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141607+0000", 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9262e-05 2024-04-18T21:51:47.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141676+0000", 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8579999999999997e-05 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.117300910999999, 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50698967299999997, 2024-04-18T21:51:47.869 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 34 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735116+0000", 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735123+0000", 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3629999999999996e-06 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735156+0000", 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3030000000000001e-05 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735167+0000", 2024-04-18T21:51:47.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.13e-05 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939428+0000", 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20426096299999999 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939615+0000", 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018654199999999999 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939841+0000", 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022621800000000001 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141747+0000", 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201905318 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.241958+0000", 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100211063 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242041+0000", 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3300000000000005e-05 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242105+0000", 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3895999999999999e-05 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.117225912, 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61088117099999995, 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 35 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.876 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735191+0000", 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3699999999999999e-07 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735266+0000", 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4856000000000003e-05 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735279+0000", 2024-04-18T21:51:47.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.36e-05 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735288+0000", 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0860000000000008e-06 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242298+0000", 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50700964999999998 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242568+0000", 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027023100000000001 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242974+0000", 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040533399999999998 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246576+0000", 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036026919999999998 2024-04-18T21:51:47.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345792+0000", 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099215503999999996 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345953+0000", 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161189 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346071+0000", 2024-04-18T21:51:47.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118192 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.117113419000001, 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61108955600000003, 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.882 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 36 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.23e-07 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735310+0000", 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7560000000000002e-06 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735323+0000", 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735337+0000", 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4049999999999999e-05 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244243+0000", 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50890638099999996 2024-04-18T21:51:47.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244422+0000", 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000178495 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244638+0000", 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216562 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.247950+0000", 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033121750000000001 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346232+0000", 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098281564000000002 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346324+0000", 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2042999999999999e-05 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346392+0000", 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8052999999999994e-05 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.1170644, 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71224671699999997, 2024-04-18T21:51:47.888 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 37 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0999999999999998e-07 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735358+0000", 2024-04-18T21:51:47.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2639999999999999e-06 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735367+0000", 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735375+0000", 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5360000000000004e-06 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245345+0000", 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50996983699999998 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245615+0000", 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027018099999999998 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246022+0000", 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000407394 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.248536+0000", 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025132330000000001 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447444+0000", 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198908419 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447532+0000", 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8010000000000006e-05 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447598+0000", 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6383000000000005e-05 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.48752445, 2024-04-18T21:51:47.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.404608044, 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364893+0000", 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1429999999999999e-06 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364969+0000", 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6136000000000004e-05 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364981+0000", 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2099999999999999e-05 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465122+0000", 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10014120999999999 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465287+0000", 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000164499 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465554+0000", 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000267194 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465927+0000", 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000373171 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.467760+0000", 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018333749999999999 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769354+0000", 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.301593271 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769435+0000", 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1433000000000005e-05 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769500+0000", 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4511999999999995e-05 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:47.900 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.469033717, 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40557132600000001, 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 85 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383383+0000", 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1799999999999999e-06 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383389+0000", 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6030000000000004e-06 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383400+0000", 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.059e-05 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383410+0000", 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.039e-05 2024-04-18T21:51:47.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.485426+0000", 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10201608199999999 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.485677+0000", 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025043800000000002 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486126+0000", 2024-04-18T21:51:47.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00044914 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489043+0000", 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029174309999999998 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788775+0000", 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29973160999999998 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788881+0000", 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106443 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788954+0000", 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2418999999999999e-05 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:47.906 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.468993981000001, 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40574237800000001, 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 86 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383423+0000", 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4700000000000001e-07 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383428+0000", 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383437+0000", 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4630000000000007e-06 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383445+0000", 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.994e-06 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486731+0000", 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103285425 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486864+0000", 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013336899999999999 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487072+0000", 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020769500000000001 2024-04-18T21:51:47.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489073+0000", 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020010010000000001 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789030+0000", 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29995712499999999 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789104+0000", 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4257000000000004e-05 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789164+0000", 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0291999999999998e-05 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:47.912 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.468954428, 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40594649700000002, 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 87 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5400000000000001e-07 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383535+0000", 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2879000000000005e-05 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383546+0000", 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1362999999999999e-05 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483681+0000", 2024-04-18T21:51:47.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100134521 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487677+0000", 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039962119999999999 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487809+0000", 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013202900000000001 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.488004+0000", 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000195198 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.490235+0000", 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002230073 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789259+0000", 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29902419899999999 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789354+0000", 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5443000000000001e-05 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789408+0000", 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3925999999999998e-05 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.918 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.368697300999999, 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40540477699999999, 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 88 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483720+0000", 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5859999999999999e-06 2024-04-18T21:51:47.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483731+0000", 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1090000000000001e-05 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483753+0000", 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1426000000000001e-05 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483765+0000", 2024-04-18T21:51:47.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2013e-05 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489152+0000", 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053867120000000001 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489297+0000", 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014540199999999999 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489556+0000", 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025916099999999999 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.490845+0000", 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00128853 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.888963+0000", 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39811801499999999 2024-04-18T21:51:47.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.889050+0000", 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7583000000000005e-05 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.889123+0000", 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3258999999999998e-05 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.203663886999999, 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61732750999999997, 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 117 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.925 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648753+0000", 2024-04-18T21:51:47.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648758+0000", 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648768+0000", 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.414e-06 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648777+0000", 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.363e-06 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863479+0000", 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214701524 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863596+0000", 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117039 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863810+0000", 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021416800000000001 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.864846+0000", 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001036319 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265892+0000", 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40104641899999999 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265995+0000", 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010209600000000001 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.266080+0000", 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5049000000000005e-05 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:47.930 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.5467526469999999, 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.509317829, 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 158 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:47.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305665+0000", 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5230000000000001e-06 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305786+0000", 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012069 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305805+0000", 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9836000000000001e-05 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305821+0000", 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6030000000000001e-05 2024-04-18T21:51:47.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307593+0000", 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017715299999999999 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307838+0000", 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024551800000000002 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.308174+0000", 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033593400000000003 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413731+0000", 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105556865 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814825+0000", 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40109329500000002 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814911+0000", 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6138999999999996e-05 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814981+0000", 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0469000000000006e-05 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:47.936 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.5465675909999996, 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50938204099999995, 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 159 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305849+0000", 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0559999999999999e-06 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305856+0000", 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9600000000000003e-06 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305867+0000", 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0910000000000001e-05 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305879+0000", 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.147e-05 2024-04-18T21:51:47.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309071+0000", 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031920859999999998 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309303+0000", 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023236200000000001 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309671+0000", 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036798000000000001 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416908+0000", 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107236578 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815085+0000", 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39817670700000002 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815165+0000", 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0196000000000003e-05 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815230+0000", 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5735999999999995e-05 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.942 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.5465242879999996, 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50958771300000005, 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 160 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:47.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305898+0000", 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1359999999999996e-06 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305907+0000", 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5300000000000002e-06 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305915+0000", 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1869999999999997e-06 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310667+0000", 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047516800000000003 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310895+0000", 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000227875 2024-04-18T21:51:47.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.311257+0000", 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000361737 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416933+0000", 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10567686699999999 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815344+0000", 2024-04-18T21:51:47.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39841079899999998 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815421+0000", 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7229000000000002e-05 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815479+0000", 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7973000000000002e-05 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.948 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.5464896259999996, 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60955713600000005, 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 161 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.949 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305927+0000", 2024-04-18T21:51:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306056+0000", 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012895300000000001 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306068+0000", 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306077+0000", 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9430000000000006e-06 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412179+0000", 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106101709 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412267+0000", 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8251999999999995e-05 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412388+0000", 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121136 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420472+0000", 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080833510000000008 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915141+0000", 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49466944899999998 2024-04-18T21:51:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915356+0000", 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215305 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915484+0000", 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012713500000000001 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.5463258030000002, 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61031149600000001, 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 162 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.955 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306091+0000", 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306097+0000", 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6300000000000003e-06 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306106+0000", 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2199999999999998e-06 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306116+0000", 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8870000000000004e-06 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413869+0000", 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107753534 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414135+0000", 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026615200000000002 2024-04-18T21:51:47.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414518+0000", 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038217299999999997 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420492+0000", 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059740879999999998 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915940+0000", 2024-04-18T21:51:47.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49544827600000002 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916143+0000", 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000202752 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916402+0000", 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025901100000000002 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.5462894330000001, 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61064147300000005, 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 163 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.961 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.06e-07 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306136+0000", 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6999999999999997e-06 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306146+0000", 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8439999999999993e-06 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306154+0000", 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415593+0000", 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109439392 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415833+0000", 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000240101 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416223+0000", 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000389804 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420508+0000", 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042851699999999996 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916537+0000", 2024-04-18T21:51:47.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49602917499999999 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916669+0000", 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179899999999999 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916768+0000", 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8879e-05 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.5462509000000004, 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61096294100000004, 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 164 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.967 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306166+0000", 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:51:47.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306275+0000", 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109749 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306287+0000", 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1182999999999999e-05 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.969 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306298+0000", 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1487e-05 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417044+0000", 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110745818 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417282+0000", 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000237712 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417665+0000", 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000383194 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420523+0000", 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028578010000000001 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916880+0000", 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49635731900000002 2024-04-18T21:51:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917054+0000", 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017436200000000001 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917128+0000", 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3693000000000001e-05 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.546104905, 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70940268200000001, 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 165 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306312+0000", 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:51:47.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306317+0000", 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0699999999999997e-06 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306327+0000", 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7030000000000001e-06 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306349+0000", 2024-04-18T21:51:47.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2160000000000001e-05 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419233+0000", 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112884159 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419499+0000", 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265865 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419851+0000", 2024-04-18T21:51:47.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035253999999999998 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.421040+0000", 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011886380000000001 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015628+0000", 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59458825299999996 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:47.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015677+0000", 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8470000000000002e-05 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015714+0000", 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7123999999999999e-05 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:47.978 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:47.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-18T21:51:47.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:51:47.981 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:47.980+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:48.081 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:48.079+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:48.113 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:118 3.17s0 3.83ece5b7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.51329150700000004, 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51331134099999998, 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:48.114 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 118 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604693+0000", 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2970000000000002e-06 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.605995+0000", 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001302276 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606021+0000", 2024-04-18T21:51:48.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5320000000000002e-05 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606036+0000", 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5133000000000001e-05 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606232+0000", 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000196672 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606237+0000", 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7199999999999997e-06 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:119 3.17s0 3.83ece5b7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.51190965799999999, 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51197241999999998, 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.118 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 119 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:51:48.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606074+0000", 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.257e-06 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606080+0000", 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0100000000000001e-06 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606095+0000", 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5413e-05 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606104+0000", 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0899999999999994e-06 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606286+0000", 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000181552 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606287+0000", 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.026e-06 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:120 3.17s0 3.83ece5b7 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.51184052800000002, 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51193898900000001, 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 120 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.123 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606143+0000", 2024-04-18T21:51:48.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.16e-07 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606148+0000", 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4569999999999996e-06 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606163+0000", 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503e-05 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606175+0000", 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2036e-05 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706293+0000", 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100117875 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706297+0000", 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.126e-06 2024-04-18T21:51:48.126 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.127 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.127 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.127 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.127 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:51:48.127 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-04-18T21:51:48.127 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:48.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:51:48.182 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:48.179+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:48.250 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:48.251 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:48.251 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:51:48.251 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:51:48.251 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:48.265 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:51:48.309 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:48.312+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:48.384 INFO:tasks.rados.rados.0.smithi097.stdout:33: finishing write tid 1 to smithi09731216-33 2024-04-18T21:51:48.384 INFO:tasks.rados.rados.0.smithi097.stdout:33: finishing write tid 2 to smithi09731216-33 2024-04-18T21:51:48.385 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:48.384+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:48.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:51:48.396 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:48.396 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:51:48.396 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:51:48.396 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:48.396 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.396 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es4 30/26 ECSubWrite(tid=9, reqid=client.4509.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.343138+0000", 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.0553054409999998, 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017587119999999999, 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343138+0000", 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.397 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343138+0000", 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343140+0000", 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2630000000000002e-06 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343170+0000", 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9569000000000001e-05 2024-04-18T21:51:48.398 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343211+0000", 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1743999999999999e-05 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343237+0000", 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5150000000000001e-05 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.399 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343421+0000", 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018436400000000001 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343448+0000", 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7453999999999998e-05 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.344818+0000", 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013701290000000001 2024-04-18T21:51:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.344897+0000", 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8039000000000003e-05 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 30/26 ECSubWrite(tid=16, reqid=client.4509.0:80, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.361490+0000", 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.0369528840000002, 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100966898, 2024-04-18T21:51:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361490+0000", 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361490+0000", 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361492+0000", 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0030000000000001e-06 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361513+0000", 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0937e-05 2024-04-18T21:51:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361546+0000", 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2792e-05 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361569+0000", 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2986999999999999e-05 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.362233+0000", 2024-04-18T21:51:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00066352800000000003 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.362261+0000", 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8030000000000001e-05 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.462388+0000", 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100127678 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.462457+0000", 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8942999999999994e-05 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.cs1 30/26 ECSubRead(tid=17, to_read={3:3bc99ad3:::smithi09731216-27:head=0,4096,0}, subchunks={3:3bc99ad3:::smithi09731216-27:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.361903+0000", 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.036539984, 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100478295, 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361903+0000", 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361903+0000", 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361905+0000", 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361919+0000", 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4459e-05 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361964+0000", 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5052999999999997e-05 2024-04-18T21:51:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361989+0000", 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5247000000000001e-05 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.461894+0000", 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099904973999999994 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.462382+0000", 2024-04-18T21:51:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00048726200000000002 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.fs7 30/26 ECSubWrite(tid=143, reqid=client.4509.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=30'4))", 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.451119+0000", 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.947324587, 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012304340000000001, 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.451119+0000", 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.451119+0000", 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.451120+0000", 2024-04-18T21:51:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.807e-06 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.451141+0000", 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0950000000000001e-05 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.451173+0000", 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2045999999999998e-05 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.451199+0000", 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5933000000000001e-05 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.451388+0000", 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018905199999999999 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.451409+0000", 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0472999999999999e-05 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.452277+0000", 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00086817700000000003 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.452349+0000", 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1996000000000007e-05 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.5s1 30/26 ECSubWrite(tid=18, reqid=client.4509.0:77, at_version=0'0, trim_to=0'0, roll_forward_to=30'4))", 2024-04-18T21:51:48.414 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.452327+0000", 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.9461158429999998, 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10257228, 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.452327+0000", 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.415 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.452327+0000", 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.452329+0000", 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9860000000000001e-06 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.452357+0000", 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7929999999999999e-05 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.416 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.452395+0000", 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8157000000000001e-05 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.452420+0000", 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4966000000000001e-05 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.553792+0000", 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101372033 2024-04-18T21:51:48.417 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.553807+0000", 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4396000000000001e-05 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.554834+0000", 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010274590000000001 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.554900+0000", 2024-04-18T21:51:48.418 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5352999999999996e-05 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.5s1 30/26 ECSubRead(tid=19, to_read={3:ab33d1b5:::smithi09731216-26:head=0,4096,0}, subchunks={3:ab33d1b5:::smithi09731216-26:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.453479+0000", 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.9449642890000001, 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10120335, 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.419 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.453479+0000", 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.453479+0000", 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.453480+0000", 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.420 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.453496+0000", 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5630000000000001e-05 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.453525+0000", 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9312999999999999e-05 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.453542+0000", 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.677e-05 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.554182+0000", 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10063991799999999 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.554682+0000", 2024-04-18T21:51:48.422 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050041899999999999 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=7, reqid=client.4509.0:94, at_version=30'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.563588+0000", 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.834854835, 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023907640000000001, 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.423 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.563588+0000", 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.563588+0000", 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.563590+0000", 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.813e-06 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.424 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.563823+0000", 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000232339 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.563854+0000", 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0939000000000003e-05 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.563870+0000", 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6796999999999999e-05 2024-04-18T21:51:48.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.564042+0000", 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017120199999999999 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.564071+0000", 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.959e-05 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.565889+0000", 2024-04-18T21:51:48.426 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018175610000000001 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.565979+0000", 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0523e-05 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.831995711, 2024-04-18T21:51:48.427 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201960743, 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.428 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566449+0000", 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.77e-06 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566495+0000", 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5488999999999998e-05 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566536+0000", 2024-04-18T21:51:48.429 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0756000000000001e-05 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.666638+0000", 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102466 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766859+0000", 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10022078500000001 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.430 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766875+0000", 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5806000000000001e-05 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768312+0000", 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014373389999999999 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768408+0000", 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6331999999999999e-05 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.431 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 30/26 ECSubWrite(tid=11, reqid=client.4509.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.672343+0000", 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.7261001729999998, 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001897485, 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.672343+0000", 2024-04-18T21:51:48.432 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.672343+0000", 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.672345+0000", 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8470000000000001e-06 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.672390+0000", 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5466000000000001e-05 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.672436+0000", 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5713000000000001e-05 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.672461+0000", 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4944e-05 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.672632+0000", 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017071400000000001 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.672662+0000", 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0216999999999999e-05 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.674142+0000", 2024-04-18T21:51:48.435 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014804110000000001 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.674241+0000", 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8172999999999997e-05 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.4s4 30/26 ECSubWrite(tid=10, reqid=client.4509.0:97, at_version=30'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.673533+0000", 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.7249105490000001, 2024-04-18T21:51:48.436 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068126810000000001, 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673533+0000", 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673533+0000", 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673535+0000", 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8470000000000001e-06 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673786+0000", 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025133799999999998 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673824+0000", 2024-04-18T21:51:48.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7905999999999999e-05 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673848+0000", 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.421e-05 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677321+0000", 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034731279999999998 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.439 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677344+0000", 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2906999999999999e-05 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680260+0000", 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029155190000000001 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680345+0000", 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5826000000000001e-05 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.440 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.4s4 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.721468974, 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19094267400000001, 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:51:48.441 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676976+0000", 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9e-06 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.442 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677025+0000", 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8730000000000003e-05 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677075+0000", 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0405999999999998e-05 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677101+0000", 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5777000000000001e-05 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.443 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.774116+0000", 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.097015016999999995 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.774145+0000", 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9009999999999998e-05 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867877+0000", 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093731420999999995 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867917+0000", 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0413000000000001e-05 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 30/26 ECSubWrite(tid=20, reqid=client.4509.0:100, at_version=30'7, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.765939+0000", 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.6325039210000001, 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101923558, 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.445 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.765939+0000", 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.765939+0000", 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.765941+0000", 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9400000000000001e-06 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766287+0000", 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00034620700000000001 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766321+0000", 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3853000000000003e-05 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766339+0000", 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7476999999999999e-05 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766475+0000", 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013571500000000001 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766489+0000", 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.489e-05 2024-04-18T21:51:48.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867806+0000", 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10131647000000001 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867863+0000", 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7006000000000003e-05 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.767094+0000", 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.6313488440000001, 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10146237900000001, 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767094+0000", 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767094+0000", 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767096+0000", 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5770000000000001e-06 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767136+0000", 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0173000000000003e-05 2024-04-18T21:51:48.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767165+0000", 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9023e-05 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767189+0000", 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3456999999999999e-05 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867518+0000", 2024-04-18T21:51:48.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10032908 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867532+0000", 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4547e-05 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.868505+0000", 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097289200000000005 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.868557+0000", 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1629999999999999e-05 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=9, reqid=client.4509.0:95, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.969491+0000", 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.4289519470000001, 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102384037, 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969491+0000", 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969491+0000", 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.455 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969493+0000", 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.823e-06 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969523+0000", 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.029e-05 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969917+0000", 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000393574 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969963+0000", 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6085999999999999e-05 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970142+0000", 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000178822 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970169+0000", 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6889999999999998e-05 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.071875+0000", 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10170655200000001 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.0s5 30/26 ECSubRead(tid=10, to_read={3:033c99c3:::smithi09731216-32:head=0,4096,0}, subchunks={3:033c99c3:::smithi09731216-32:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.976905+0000", 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.421538637, 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0054790120000000001, 2024-04-18T21:51:48.458 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.976905+0000", 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.976905+0000", 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.976906+0000", 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.747e-06 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.976930+0000", 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3246000000000001e-05 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.976962+0000", 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2196999999999997e-05 2024-04-18T21:51:48.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.976993+0000", 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0806000000000003e-05 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.982319+0000", 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053267870000000004 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.982384+0000", 2024-04-18T21:51:48.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4228999999999998e-05 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 31/26 ECSubWrite(tid=22, reqid=client.4509.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=30'8))", 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.274149+0000", 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.124294605, 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101433601, 2024-04-18T21:51:48.462 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274149+0000", 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274149+0000", 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274151+0000", 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0229999999999999e-06 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274178+0000", 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7347000000000001e-05 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274222+0000", 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4016e-05 2024-04-18T21:51:48.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274249+0000", 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7046000000000002e-05 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.374563+0000", 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100314194 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.374578+0000", 2024-04-18T21:51:48.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.519e-05 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.375509+0000", 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093072900000000004 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.375582+0000", 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3056e-05 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 31/26 ECSubWrite(tid=14, reqid=client.4509.0:115, at_version=31'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.277913+0000", 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.1205298960000001, 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027106750000000001, 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.277913+0000", 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.277913+0000", 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.277915+0000", 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9970000000000001e-06 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.278261+0000", 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00034569700000000001 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.278303+0000", 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1903000000000002e-05 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.278331+0000", 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7786000000000001e-05 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.278507+0000", 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175809 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.278538+0000", 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1285999999999998e-05 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280560+0000", 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020217310000000001 2024-04-18T21:51:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280624+0000", 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4466000000000002e-05 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 31/26 ECSubWrite(tid=15, reqid=client.4509.0:116, at_version=31'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.280741+0000", 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.117702642, 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101446435, 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.471 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280741+0000", 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280741+0000", 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280742+0000", 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280783+0000", 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1112999999999998e-05 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280822+0000", 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8537000000000001e-05 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280845+0000", 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3399000000000001e-05 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.381044+0000", 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100199002 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.381063+0000", 2024-04-18T21:51:48.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8502999999999999e-05 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.382093+0000", 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010302320000000001 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.382187+0000", 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3995999999999999e-05 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.cs1 31/26 ECSubRead(tid=23, to_read={3:348e44d8:::smithi09731216-34:head=0,4096,0}, subchunks={3:348e44d8:::smithi09731216-34:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.376949+0000", 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.021494401, 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10074501299999999, 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.376949+0000", 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.376949+0000", 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.376951+0000", 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.807e-06 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.376968+0000", 2024-04-18T21:51:48.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7686e-05 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.377002+0000", 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3606000000000003e-05 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.377023+0000", 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1310000000000001e-05 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.477292+0000", 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10026877100000001 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.477694+0000", 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00040183299999999999 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 31/26 ECSubWrite(tid=16, reqid=client.4509.0:116, at_version=0'0, trim_to=0'0, roll_forward_to=31'6))", 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.404861+0000", 2024-04-18T21:51:48.479 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.99358269499999996, 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001513464, 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.404861+0000", 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.404861+0000", 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.404863+0000", 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0430000000000002e-06 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.404890+0000", 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.773e-05 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.404928+0000", 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7313e-05 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.404951+0000", 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2972999999999998e-05 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405083+0000", 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131989 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405111+0000", 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8717e-05 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.406309+0000", 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011972899999999999 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.406374+0000", 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5408999999999997e-05 2024-04-18T21:51:48.483 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.484 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.484 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.484 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:48.485 INFO:tasks.rados.rados.0.smithi097.stdout:36: finishing write tid 1 to smithi09731216-36 2024-04-18T21:51:48.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:51:48.486 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:48.488+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:48.487 INFO:tasks.rados.rados.0.smithi097.stdout:33: finishing write tid 3 to smithi09731216-33 2024-04-18T21:51:48.487 INFO:tasks.rados.rados.0.smithi097.stdout:33: oid 33 updating version 0 to 1 2024-04-18T21:51:48.487 INFO:tasks.rados.rados.0.smithi097.stdout:33: oid 33 updating version 1 to 2 2024-04-18T21:51:48.487 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 33 v 2 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-18T21:51:48.487 INFO:tasks.rados.rados.0.smithi097.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-18T21:51:48.488 INFO:tasks.rados.rados.0.smithi097.stdout:33: done (15 left) 2024-04-18T21:51:48.488 INFO:tasks.rados.rados.0.smithi097.stdout:42: write initial oid 42 2024-04-18T21:51:48.488 INFO:tasks.rados.rados.0.smithi097.stdout:42: seq_num 41 ranges {0=704512} 2024-04-18T21:51:48.499 INFO:tasks.rados.rados.0.smithi097.stdout:42: writing smithi09731216-42 from 0 to 704512 tid 1 2024-04-18T21:51:48.499 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:48.586 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:48.583+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:48.590 INFO:tasks.rados.rados.0.smithi097.stdout:35: finishing write tid 1 to smithi09731216-35 2024-04-18T21:51:48.644 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:51:48.644 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.17s4 32/31 ECSubWrite(tid=1, reqid=client.4509.0:119, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.172960064, 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17300280800000001, 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.664 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495130+0000", 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0860000000000001e-06 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495154+0000", 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4457999999999999e-05 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495174+0000", 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9426999999999999e-05 2024-04-18T21:51:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495188+0000", 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4813e-05 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.14s6 32/31 ECSubWrite(tid=20, reqid=client.4509.0:125, at_version=32'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.071583913999999998, 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.071714727000000006, 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.667 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3799999999999997e-07 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596521+0000", 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5455999999999999e-05 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596532+0000", 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0881e-05 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.670 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:51:48.670 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 2 2024-04-18T21:51:48.670 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:48.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:51:48.687 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:48.683+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:48.696 INFO:tasks.rados.rados.0.smithi097.stdout:35: finishing write tid 2 to smithi09731216-35 2024-04-18T21:51:48.718 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:51:48.718 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:51:48.738 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:51:48.738 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:51:48.788 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:48.788+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:48.790 INFO:tasks.rados.rados.0.smithi097.stdout:40: finishing write tid 1 to smithi09731216-40 2024-04-18T21:51:48.790 INFO:tasks.rados.rados.0.smithi097.stdout:40: finishing write tid 2 to smithi09731216-40 2024-04-18T21:51:48.796 INFO:tasks.rados.rados.0.smithi097.stdout:32: finishing write tid 3 to smithi09731216-32 2024-04-18T21:51:48.796 INFO:tasks.rados.rados.0.smithi097.stdout:32: oid 32 updating version 0 to 5 2024-04-18T21:51:48.796 INFO:tasks.rados.rados.0.smithi097.stdout:32: oid 32 updating version 5 to 6 2024-04-18T21:51:48.796 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-18T21:51:48.796 INFO:tasks.rados.rados.0.smithi097.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:51:48.796 INFO:tasks.rados.rados.0.smithi097.stdout:32: done (15 left) 2024-04-18T21:51:48.796 INFO:tasks.rados.rados.0.smithi097.stdout:43: write initial oid 43 2024-04-18T21:51:48.796 INFO:tasks.rados.rados.0.smithi097.stdout:43: seq_num 42 ranges {0=720896} 2024-04-18T21:51:48.808 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:48.808 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:48.808 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:51:48.808 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:51:48.808 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:48.809 INFO:tasks.rados.rados.0.smithi097.stdout:43: writing smithi09731216-43 from 0 to 720896 tid 1 2024-04-18T21:51:48.809 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:48.809 INFO:tasks.rados.rados.0.smithi097.stdout:42: finishing write tid 1 to smithi09731216-42 2024-04-18T21:51:48.809 INFO:tasks.rados.rados.0.smithi097.stdout:42: finishing write tid 2 to smithi09731216-42 2024-04-18T21:51:48.809 INFO:tasks.rados.rados.0.smithi097.stdout:25: finishing write tid 1 to smithi09731216-25 2024-04-18T21:51:48.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:51:48.888 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:48.883+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:48.895 INFO:tasks.rados.rados.0.smithi097.stdout:35: finishing write tid 3 to smithi09731216-35 2024-04-18T21:51:48.895 INFO:tasks.rados.rados.0.smithi097.stdout:35: oid 35 updating version 0 to 1 2024-04-18T21:51:48.895 INFO:tasks.rados.rados.0.smithi097.stdout:35: oid 35 updating version 1 to 2 2024-04-18T21:51:48.895 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 35 v 2 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-18T21:51:48.896 INFO:tasks.rados.rados.0.smithi097.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-18T21:51:48.896 INFO:tasks.rados.rados.0.smithi097.stdout:35: done (15 left) 2024-04-18T21:51:48.896 INFO:tasks.rados.rados.0.smithi097.stdout:44: write initial oid 44 2024-04-18T21:51:48.896 INFO:tasks.rados.rados.0.smithi097.stdout:44: seq_num 43 ranges {0=737280} 2024-04-18T21:51:48.906 INFO:tasks.rados.rados.0.smithi097.stdout:44: writing smithi09731216-44 from 0 to 737280 tid 1 2024-04-18T21:51:48.906 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:48.906 INFO:tasks.rados.rados.0.smithi097.stdout:25: finishing write tid 2 to smithi09731216-25 2024-04-18T21:51:48.929 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:51:48.929 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:51:48.977 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is unset 2024-04-18T21:51:48.989 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:48.983+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:48.992 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:51:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es6 30/26 ECSubWrite(tid=9, reqid=client.4509.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.343173+0000", 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.6524456020000002, 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00160064, 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343173+0000", 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343173+0000", 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343175+0000", 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0499999999999999e-06 2024-04-18T21:51:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343201+0000", 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5190000000000001e-05 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343237+0000", 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6786000000000003e-05 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343261+0000", 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3260000000000001e-05 2024-04-18T21:51:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343433+0000", 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017244800000000001 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.343457+0000", 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3646999999999999e-05 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.344665+0000", 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012084229999999999 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.344774+0000", 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108836 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.9s5 30/26 ECSubWrite(tid=9, reqid=client.4509.0:68, at_version=0'0, trim_to=0'0, roll_forward_to=29'6))", 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.351650+0000", 2024-04-18T21:51:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.643969271, 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017376049999999999, 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.351650+0000", 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.351650+0000", 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.351652+0000", 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8360000000000001e-06 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.351673+0000", 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1843000000000001e-05 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.351704+0000", 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.025e-05 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.351722+0000", 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7969999999999999e-05 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.351853+0000", 2024-04-18T21:51:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013132500000000001 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.351875+0000", 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2177000000000001e-05 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.353353+0000", 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001477681 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.353387+0000", 2024-04-18T21:51:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4523000000000001e-05 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.9s5 30/26 ECSubRead(tid=10, to_read={3:9da3af1e:::smithi09731216-23:head=0,4096,0}, subchunks={3:9da3af1e:::smithi09731216-23:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.352738+0000", 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.6428807770000002, 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00065319800000000002, 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.352738+0000", 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.352738+0000", 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.352740+0000", 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7129999999999999e-06 2024-04-18T21:51:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.352756+0000", 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5974e-05 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.352784+0000", 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8221999999999998e-05 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.352811+0000", 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6593000000000001e-05 2024-04-18T21:51:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.352968+0000", 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015754300000000001 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.353391+0000", 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042315300000000001 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs6 30/26 ECSubWrite(tid=16, reqid=client.4509.0:80, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.361452+0000", 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.6341669950000002, 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014585379999999999, 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361452+0000", 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361452+0000", 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361454+0000", 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8640000000000001e-06 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361477+0000", 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3589999999999999e-05 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361507+0000", 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9796000000000001e-05 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361523+0000", 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5840000000000001e-05 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361679+0000", 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000155542 2024-04-18T21:51:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.361703+0000", 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4323000000000001e-05 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.362847+0000", 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011436040000000001 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.362911+0000", 2024-04-18T21:51:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3979000000000005e-05 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=7, reqid=client.4509.0:94, at_version=30'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.334544562, 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201169337, 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661077+0000", 2024-04-18T21:51:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4600000000000002e-06 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661394+0000", 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031720499999999998 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661424+0000", 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9822999999999999e-05 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661441+0000", 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7496e-05 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661661+0000", 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000219602 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661676+0000", 2024-04-18T21:51:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5333e-05 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862187+0000", 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200510299 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862244+0000", 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7119e-05 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.334141255, 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201435581, 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661479+0000", 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0839999999999999e-06 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661509+0000", 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0389999999999999e-05 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661529+0000", 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0219000000000001e-05 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661553+0000", 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3836999999999999e-05 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861851+0000", 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20029786299999999 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861866+0000", 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4769999999999999e-05 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862836+0000", 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097006600000000001 2024-04-18T21:51:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862913+0000", 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7352000000000001e-05 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.4s6 30/26 ECSubWrite(tid=10, reqid=client.4509.0:97, at_version=30'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.673684+0000", 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.3219353269999998, 2024-04-18T21:51:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034327390000000002, 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673684+0000", 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673684+0000", 2024-04-18T21:51:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673685+0000", 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.113e-06 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.674069+0000", 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00038428099999999999 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.674113+0000", 2024-04-18T21:51:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3506e-05 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.674138+0000", 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5406e-05 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.674358+0000", 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000220122 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.674382+0000", 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3649e-05 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677036+0000", 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026543819999999998 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677116+0000", 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0279999999999997e-05 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.4s6 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.677102+0000", 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.318516518, 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002542402, 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677102+0000", 2024-04-18T21:51:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677102+0000", 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677104+0000", 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5230000000000001e-06 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677147+0000", 2024-04-18T21:51:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2753000000000002e-05 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677175+0000", 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8563000000000001e-05 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677199+0000", 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3649999999999999e-05 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677439+0000", 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024018100000000001 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677463+0000", 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3856999999999999e-05 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.679573+0000", 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021104460000000002 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.679645+0000", 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1428999999999997e-05 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs6 30/26 ECSubWrite(tid=20, reqid=client.4509.0:100, at_version=30'7, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.766112+0000", 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.229507103, 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029180170000000002, 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766112+0000", 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766112+0000", 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766114+0000", 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.34e-06 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766398+0000", 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028387799999999999 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766430+0000", 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1609999999999997e-05 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766447+0000", 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.698e-05 2024-04-18T21:51:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766633+0000", 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018620400000000001 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766662+0000", 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8929999999999999e-05 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768970+0000", 2024-04-18T21:51:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023077649999999998 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.769030+0000", 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0309999999999997e-05 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs6 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.767175+0000", 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.228443688, 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024094239999999999, 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767175+0000", 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767175+0000", 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767177+0000", 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.587e-06 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767218+0000", 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1155999999999997e-05 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767270+0000", 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1946e-05 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.767298+0000", 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7627000000000002e-05 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768075+0000", 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00077766700000000001 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768104+0000", 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9153e-05 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.769533+0000", 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001428712 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.769585+0000", 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1576000000000002e-05 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.8s4 30/26 ECSubWrite(tid=11, reqid=client.4509.0:103, at_version=30'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.771551+0000", 2024-04-18T21:51:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.2240679800000001, 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090780761000000001, 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.771551+0000", 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.771551+0000", 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.771553+0000", 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.84e-06 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.771904+0000", 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035079399999999998 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.771934+0000", 2024-04-18T21:51:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0710000000000002e-05 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.771951+0000", 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6762999999999999e-05 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.772074+0000", 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122755 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.772087+0000", 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3343000000000001e-05 2024-04-18T21:51:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862262+0000", 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090174939999999995 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862332+0000", 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9616000000000006e-05 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.8s4 30/26 ECSubWrite(tid=12, reqid=client.4509.0:104, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.1260609540000002, 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11605781900000001, 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869560+0000", 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-06 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869593+0000", 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3216000000000002e-05 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869624+0000", 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1164000000000002e-05 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969744+0000", 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100119518 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969958+0000", 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021381800000000001 2024-04-18T21:51:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969985+0000", 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7376999999999999e-05 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985510+0000", 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015525357 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985616+0000", 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105479 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=9, reqid=client.4509.0:95, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.970339+0000", 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.0252798539999999, 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016383587000000002, 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970339+0000", 2024-04-18T21:51:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970339+0000", 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970341+0000", 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.99e-06 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970370+0000", 2024-04-18T21:51:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.881e-05 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970411+0000", 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0812999999999997e-05 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970434+0000", 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3359000000000001e-05 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970557+0000", 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012266600000000001 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.970579+0000", 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1970000000000001e-05 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.986617+0000", 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016037862999999999 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.986723+0000", 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106116 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs6 31/26 ECSubWrite(tid=22, reqid=client.4509.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=30'8))", 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.274243+0000", 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.72137579, 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001829352, 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274243+0000", 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274243+0000", 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274244+0000", 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7700000000000004e-07 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274266+0000", 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.198e-05 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274311+0000", 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5256000000000002e-05 2024-04-18T21:51:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274337+0000", 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5780000000000001e-05 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274497+0000", 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000159875 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274522+0000", 2024-04-18T21:51:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5647e-05 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275975+0000", 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001452385 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.276073+0000", 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7652000000000006e-05 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.17s4 32/31 ECSubWrite(tid=0, reqid=client.4509.0:118, at_version=32'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.490156+0000", 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.50546322300000002, 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095582515000000007, 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490156+0000", 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490156+0000", 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490157+0000", 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.068e-06 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490329+0000", 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000171998 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490349+0000", 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0352000000000001e-05 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490362+0000", 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2826000000000001e-05 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583484+0000", 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093122387000000001 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583498+0000", 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3161e-05 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.585697+0000", 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021995320000000001 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.585738+0000", 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1190999999999999e-05 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.17s4 32/31 ECSubWrite(tid=1, reqid=client.4509.0:119, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.50049039500000003, 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18978218799999999, 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:51:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495130+0000", 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0860000000000001e-06 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495154+0000", 2024-04-18T21:51:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4457999999999999e-05 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495174+0000", 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9426999999999999e-05 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495188+0000", 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4813e-05 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683568+0000", 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.188379871 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683583+0000", 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5227e-05 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.684845+0000", 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012619560000000001 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.684911+0000", 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5350000000000003e-05 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.14s6 32/31 ECSubWrite(tid=19, reqid=client.4509.0:124, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.596324+0000", 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.399294547, 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026333760000000002, 2024-04-18T21:51:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596324+0000", 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596324+0000", 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596326+0000", 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1999999999999999e-06 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596444+0000", 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118396 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596462+0000", 2024-04-18T21:51:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8085999999999999e-05 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596475+0000", 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2544000000000001e-05 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596558+0000", 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3424000000000006e-05 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596571+0000", 2024-04-18T21:51:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2577e-05 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.598925+0000", 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023544019999999998 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.598958+0000", 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2746999999999997e-05 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 32/31 ECSubWrite(tid=12, reqid=client.4509.0:121, at_version=32'7, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.697178+0000", 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.29844084900000001, 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014892060000000001, 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.697178+0000", 2024-04-18T21:51:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.697178+0000", 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.697178+0000", 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.84e-07 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.697423+0000", 2024-04-18T21:51:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024465100000000003 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.697438+0000", 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4944e-05 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.697451+0000", 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3285000000000001e-05 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.697553+0000", 2024-04-18T21:51:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101166 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.697565+0000", 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2418e-05 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.698623+0000", 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010582530000000001 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.698667+0000", 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4104999999999999e-05 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 32/31 ECSubWrite(tid=13, reqid=client.4509.0:122, at_version=32'8, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.699204+0000", 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.29641479799999998, 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00065926999999999999, 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699204+0000", 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699204+0000", 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699205+0000", 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699221+0000", 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5773999999999998e-05 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699234+0000", 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3317e-05 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699241+0000", 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3200000000000002e-06 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699298+0000", 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7033000000000001e-05 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699309+0000", 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.075e-05 2024-04-18T21:51:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699830+0000", 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00052135599999999995 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.699863+0000", 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3096999999999998e-05 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.18s4 32/31 ECSubRead(tid=13, to_read={3:1b279e7b:::smithi09731216-35:head=0,4096,0}, subchunks={3:1b279e7b:::smithi09731216-35:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.701141+0000", 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.29447761900000002, 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00046577499999999998, 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.701141+0000", 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.701141+0000", 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.701142+0000", 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.58e-07 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.701150+0000", 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4249999999999998e-06 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.701162+0000", 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1812e-05 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.701171+0000", 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2699999999999993e-06 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.701236+0000", 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4494999999999998e-05 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.701607+0000", 2024-04-18T21:51:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00037131499999999999 2024-04-18T21:51:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.079 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:49.080 INFO:tasks.rados.rados.0.smithi097.stdout:41: finishing write tid 1 to smithi09731216-41 2024-04-18T21:51:49.080 INFO:tasks.rados.rados.0.smithi097.stdout:42: finishing write tid 3 to smithi09731216-42 2024-04-18T21:51:49.080 INFO:tasks.rados.rados.0.smithi097.stdout:42: oid 42 updating version 0 to 4 2024-04-18T21:51:49.080 INFO:tasks.rados.rados.0.smithi097.stdout:42: oid 42 version 4 is already newer than 3 2024-04-18T21:51:49.080 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 42 v 4 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-18T21:51:49.080 INFO:tasks.rados.rados.0.smithi097.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-18T21:51:49.080 INFO:tasks.rados.rados.0.smithi097.stdout:42: done (15 left) 2024-04-18T21:51:49.080 INFO:tasks.rados.rados.0.smithi097.stdout:45: write initial oid 45 2024-04-18T21:51:49.081 INFO:tasks.rados.rados.0.smithi097.stdout:45: seq_num 44 ranges {0=753664} 2024-04-18T21:51:49.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:45: writing smithi09731216-45 from 0 to 753664 tid 1 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:31: finishing write tid 1 to smithi09731216-31 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:31: finishing write tid 2 to smithi09731216-31 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:40: finishing write tid 3 to smithi09731216-40 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:40: oid 40 updating version 0 to 2 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:40: oid 40 version 2 is already newer than 1 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 40 v 2 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-18T21:51:49.083 INFO:tasks.rados.rados.0.smithi097.stdout:40: done (15 left) 2024-04-18T21:51:49.084 INFO:tasks.rados.rados.0.smithi097.stdout:46: write initial oid 46 2024-04-18T21:51:49.084 INFO:tasks.rados.rados.0.smithi097.stdout:46: seq_num 45 ranges {0=770048} 2024-04-18T21:51:49.084 INFO:tasks.rados.rados.0.smithi097.stdout:46: writing smithi09731216-46 from 0 to 770048 tid 1 2024-04-18T21:51:49.084 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:49.110 INFO:tasks.rados.rados.0.smithi097.stdout:41: finishing write tid 2 to smithi09731216-41 2024-04-18T21:51:49.110 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:49.107+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:49.113 INFO:tasks.rados.rados.0.smithi097.stdout:26: finishing write tid 3 to smithi09731216-26 2024-04-18T21:51:49.113 INFO:tasks.rados.rados.0.smithi097.stdout:26: oid 26 updating version 0 to 4 2024-04-18T21:51:49.113 INFO:tasks.rados.rados.0.smithi097.stdout:26: oid 26 version 4 is already newer than 3 2024-04-18T21:51:49.113 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 26 v 4 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-18T21:51:49.113 INFO:tasks.rados.rados.0.smithi097.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-18T21:51:49.113 INFO:tasks.rados.rados.0.smithi097.stdout:26: done (15 left) 2024-04-18T21:51:49.114 INFO:tasks.rados.rados.0.smithi097.stdout:47: write initial oid 47 2024-04-18T21:51:49.114 INFO:tasks.rados.rados.0.smithi097.stdout:47: seq_num 46 ranges {0=786432} 2024-04-18T21:51:49.126 INFO:tasks.rados.rados.0.smithi097.stdout:47: writing smithi09731216-47 from 0 to 786432 tid 1 2024-04-18T21:51:49.126 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:49.191 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:49.187+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.6s4 32/31 ECSubWrite(tid=11, reqid=client.4509.0:107, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.006334+0000", 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.20937303299999999, 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20942836000000001, 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006334+0000", 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006334+0000", 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006335+0000", 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3000000000000001e-07 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006370+0000", 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5309999999999999e-05 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006392+0000", 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1860000000000001e-05 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006417+0000", 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4765000000000001e-05 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206650+0000", 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20023291400000001 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206666+0000", 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6364000000000001e-05 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.215 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.16s4 32/31 ECSubWrite(tid=12, reqid=client.4509.0:109, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.20926513499999999, 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20942796, 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006443+0000", 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999998e-07 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006668+0000", 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000224878 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006685+0000", 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6730000000000001e-05 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006703+0000", 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8504000000000001e-05 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206788+0000", 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200085349 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206807+0000", 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.827e-05 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.16s4 32/31 ECSubWrite(tid=15, reqid=client.4509.0:110, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.219 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.127550937, 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12780556600000001, 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:51:49.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088157+0000", 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088207+0000", 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0584999999999998e-05 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088241+0000", 2024-04-18T21:51:49.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3784e-05 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088269+0000", 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8314999999999999e-05 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.18s1 32/31 ECSubWrite(tid=15, reqid=client.4509.0:137, at_version=32'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.093921+0000", 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.12178666, 2024-04-18T21:51:49.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.122111804, 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.093921+0000", 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.093921+0000", 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.093921+0000", 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9999999999999998e-07 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.093980+0000", 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8584e-05 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.094001+0000", 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1002e-05 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.094015+0000", 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4195e-05 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.7s3,QREQUEST,e:33,from:8(0),reservation_nonce:8)", 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.205667+0000", 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.010040713999999999, 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010432999, 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.225 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.205667+0000", 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.205667+0000", 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.205669+0000", 2024-04-18T21:51:49.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.069e-06 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.205679+0000", 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0387999999999999e-05 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.205707+0000", 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7742e-05 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.205721+0000", 2024-04-18T21:51:49.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4431e-05 2024-04-18T21:51:49.228 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.228 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.228 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.228 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.228 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:51:49.228 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 5 2024-04-18T21:51:49.228 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:49.229 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:51:49.292 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:49.287+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:49.303 INFO:tasks.rados.rados.0.smithi097.stdout:41: finishing write tid 3 to smithi09731216-41 2024-04-18T21:51:49.303 INFO:tasks.rados.rados.0.smithi097.stdout:41: oid 41 updating version 0 to 7 2024-04-18T21:51:49.303 INFO:tasks.rados.rados.0.smithi097.stdout:41: oid 41 updating version 7 to 8 2024-04-18T21:51:49.303 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 41 v 8 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-18T21:51:49.303 INFO:tasks.rados.rados.0.smithi097.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-18T21:51:49.303 INFO:tasks.rados.rados.0.smithi097.stdout:41: done (15 left) 2024-04-18T21:51:49.303 INFO:tasks.rados.rados.0.smithi097.stdout:48: write initial oid 48 2024-04-18T21:51:49.303 INFO:tasks.rados.rados.0.smithi097.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-04-18T21:51:49.310 INFO:tasks.rados.rados.0.smithi097.stdout:48: writing smithi09731216-48 from 0 to 425984 tid 1 2024-04-18T21:51:49.317 INFO:tasks.rados.rados.0.smithi097.stdout:48: writing smithi09731216-48 from 425984 to 802816 tid 2 2024-04-18T21:51:49.317 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:49.317 INFO:tasks.rados.rados.0.smithi097.stdout:43: finishing write tid 1 to smithi09731216-43 2024-04-18T21:51:49.353 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:49.354 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:49.354 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:51:49.354 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:51:49.354 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:49.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:51:49.386 INFO:tasks.rados.rados.0.smithi097.stdout:36: finishing write tid 2 to smithi09731216-36 2024-04-18T21:51:49.403 INFO:tasks.rados.rados.0.smithi097.stdout:45: finishing write tid 1 to smithi09731216-45 2024-04-18T21:51:49.403 INFO:tasks.rados.rados.0.smithi097.stdout:45: finishing write tid 2 to smithi09731216-45 2024-04-18T21:51:49.414 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:49.416+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:49.493 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:49.487+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:49.494 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:49.494 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:51:49.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:51:49.494 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:49.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.494 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:97 3.4s0 3:209f2930:::smithi09731216-33:head [append 0~540672,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:49.494 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.569059+0000", 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.925838068, 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.416082285, 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.569059+0000", 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.569059+0000", 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.569062+0000", 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2100000000000002e-06 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.570235+0000", 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011730639999999999 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.570271+0000", 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5876e-05 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670372+0000", 2024-04-18T21:51:49.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10010100199999999 2024-04-18T21:51:49.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670552+0000", 2024-04-18T21:51:49.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018013600000000001 2024-04-18T21:51:49.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670871+0000", 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031866700000000002 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673680+0000", 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028090709999999998 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.777705+0000", 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10402547299999999 2024-04-18T21:51:49.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985097+0000", 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20739194 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985141+0000", 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3846e-05 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:98 3.4s0 3:209f2930:::smithi09731216-33:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.670413+0000", 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.8244842490000002, 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41727380200000003, 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.501 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 98 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670413+0000", 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670413+0000", 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670414+0000", 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.403e-06 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670425+0000", 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0733e-05 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670443+0000", 2024-04-18T21:51:49.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8427000000000002e-05 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670455+0000", 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1327e-05 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.675388+0000", 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049329830000000002 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.675645+0000", 2024-04-18T21:51:49.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000257395 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676976+0000", 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013308389999999999 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.777778+0000", 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100801826 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087649+0000", 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30987076299999999 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087687+0000", 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8105999999999997e-05 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.506 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:99 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.670540+0000", 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.8243566840000001, 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41733552600000001, 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670540+0000", 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670540+0000", 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670541+0000", 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6700000000000002e-07 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670548+0000", 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1970000000000001e-06 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670561+0000", 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.381e-05 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670571+0000", 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3829999999999998e-06 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677632+0000", 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070607589999999998 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677871+0000", 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000239125 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087837+0000", 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40996610900000002 2024-04-18T21:51:49.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087876+0000", 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9175999999999999e-05 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.bs0 30/26 ECSubReadReply(tid=12, attrs_read=0))", 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.775303+0000", 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.7195941989999999, 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29787019399999998, 2024-04-18T21:51:49.512 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.775303+0000", 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.775303+0000", 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.775305+0000", 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7969999999999999e-06 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.775326+0000", 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.122e-05 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.775351+0000", 2024-04-18T21:51:49.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5456000000000002e-05 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.875482+0000", 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100130369 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.073099+0000", 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.197616822 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.073173+0000", 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4530000000000006e-05 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.bs0 30/26 ECSubReadReply(tid=12, attrs_read=0))", 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.965837+0000", 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.5290596189999999, 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20804524499999999, 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.516 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.965837+0000", 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.965837+0000", 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.965838+0000", 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2699999999999998e-07 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.965864+0000", 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5582999999999999e-05 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.965883+0000", 2024-04-18T21:51:49.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9490000000000001e-05 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.965903+0000", 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9173000000000001e-05 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.173535+0000", 2024-04-18T21:51:49.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.207632435 2024-04-18T21:51:49.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.173883+0000", 2024-04-18T21:51:49.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00034763699999999998 2024-04-18T21:51:49.520 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.520 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.521 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.521 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:97 3.4s0 3:209f2930:::smithi09731216-33:head [append 0~540672,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:49.521 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:51:49.521 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.5090415450000001, 2024-04-18T21:51:49.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4026191889999999, 2024-04-18T21:51:49.522 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.522 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.522 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.522 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.522 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.522 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.523 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97 2024-04-18T21:51:49.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.523 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:51:49.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:51:49.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985858+0000", 2024-04-18T21:51:49.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.92e-06 2024-04-18T21:51:49.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987540+0000", 2024-04-18T21:51:49.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001681463 2024-04-18T21:51:49.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987566+0000", 2024-04-18T21:51:49.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6043000000000001e-05 2024-04-18T21:51:49.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987586+0000", 2024-04-18T21:51:49.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9930000000000001e-05 2024-04-18T21:51:49.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087950+0000", 2024-04-18T21:51:49.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100363824 2024-04-18T21:51:49.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:49.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087956+0000", 2024-04-18T21:51:49.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9200000000000001e-06 2024-04-18T21:51:49.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388374+0000", 2024-04-18T21:51:49.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3004186340000001 2024-04-18T21:51:49.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388475+0000", 2024-04-18T21:51:49.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100455 2024-04-18T21:51:49.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.531 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.532 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.532 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:98 3.4s0 3:209f2930:::smithi09731216-33:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:49.532 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:51:49.532 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.507280143, 2024-04-18T21:51:49.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.401002842, 2024-04-18T21:51:49.533 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.533 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.533 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.533 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.533 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.533 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.533 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 98 2024-04-18T21:51:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:51:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987618+0000", 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5e-06 2024-04-18T21:51:49.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987625+0000", 2024-04-18T21:51:49.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6769999999999998e-06 2024-04-18T21:51:49.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987637+0000", 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1693e-05 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987646+0000", 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3700000000000001e-06 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088140+0000", 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10049390900000001 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088145+0000", 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3600000000000004e-06 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388530+0000", 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300384269 2024-04-18T21:51:49.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388620+0000", 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0063999999999997e-05 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:99 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.507237403, 2024-04-18T21:51:49.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503241456, 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:51:49.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987661+0000", 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5999999999999991e-07 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987664+0000", 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6459999999999999e-06 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987675+0000", 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0967e-05 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987684+0000", 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6929999999999996e-06 2024-04-18T21:51:49.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088241+0000", 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100557049 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088244+0000", 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2729999999999998e-06 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388746+0000", 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300502061 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388878+0000", 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131533 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490901+0000", 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102023274 2024-04-18T21:51:49.544 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:115 3.bs0 3:d4cd3926:::smithi09731216-39:head [append 0~638976,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.185803+0000", 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.3090938479999998, 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.21740421400000001, 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:49.545 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 115 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.185803+0000", 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.185803+0000", 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.185806+0000", 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3999999999999999e-06 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187551+0000", 2024-04-18T21:51:49.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001745843 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187585+0000", 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3572000000000003e-05 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187602+0000", 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.738e-05 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.274629+0000", 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.087026338999999994 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275035+0000", 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00040636300000000003 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.277581+0000", 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025459990000000002 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.381594+0000", 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104012824 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.403072+0000", 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.021478229000000001 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.403207+0000", 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000135265 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:116 3.bs0 3:d4cd3926:::smithi09731216-39:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.187640+0000", 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.3072569660000002, 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.21682061899999999, 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.551 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187640+0000", 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187640+0000", 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187642+0000", 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2630000000000002e-06 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187653+0000", 2024-04-18T21:51:49.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0173000000000001e-05 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187685+0000", 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2879999999999997e-05 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187703+0000", 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7805999999999999e-05 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.279354+0000", 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.091650683999999996 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.279637+0000", 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028294099999999999 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.280574+0000", 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093747700000000004 2024-04-18T21:51:49.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.381662+0000", 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10108780000000001 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.404249+0000", 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.022587079999999999 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.404461+0000", 2024-04-18T21:51:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021151500000000001 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:117 3.bs0 3:d4cd3926:::smithi09731216-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.3071683940000001, 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.520974363, 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 117 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187730+0000", 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.127e-06 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187735+0000", 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.536e-06 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187752+0000", 2024-04-18T21:51:49.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6186999999999999e-05 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187762+0000", 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0587e-05 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281120+0000", 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093357525999999996 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281296+0000", 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017659300000000001 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405205+0000", 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.123908465 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405449+0000", 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000244505 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708703+0000", 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30325383700000003 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 31/26 ECSubWriteReply(tid=16, last_complete=31'6, committed=1, applied=1))", 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.407076+0000", 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.0878208809999999, 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30124820000000002, 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.407076+0000", 2024-04-18T21:51:49.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.407076+0000", 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.407077+0000", 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.263e-06 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.407085+0000", 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1500000000000002e-06 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.407099+0000", 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4533e-05 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.507265+0000", 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100165591 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708272+0000", 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201007674 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708324+0000", 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1989e-05 2024-04-18T21:51:49.566 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 31/26 ECSubWriteReply(tid=16, last_complete=31'6, committed=1, applied=1))", 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.506464+0000", 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.98843298, 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20172807000000001, 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.506464+0000", 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.506464+0000", 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.506465+0000", 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2500000000000001e-06 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.506473+0000", 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3860000000000001e-06 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.506493+0000", 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0752999999999999e-05 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.506505+0000", 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2006e-05 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708139+0000", 2024-04-18T21:51:49.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20163333899999999 2024-04-18T21:51:49.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708192+0000", 2024-04-18T21:51:49.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3335999999999999e-05 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 31/26 ECSubWriteReply(tid=16, last_complete=31'6, committed=1, applied=1))", 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.507438+0000", 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.987459348, 2024-04-18T21:51:49.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201467339, 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.507438+0000", 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.507438+0000", 2024-04-18T21:51:49.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.507439+0000", 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5600000000000004e-07 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.507445+0000", 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3640000000000004e-06 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.507462+0000", 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7119e-05 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.507471+0000", 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1600000000000004e-06 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708793+0000", 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20132180799999999 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708905+0000", 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111932 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:124 3.14s0 3:2c76b789:::smithi09731216-42:head [append 0~704512,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.504418+0000", 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.99047891399999999, 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.298185794, 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.576 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 124 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.504418+0000", 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.504418+0000", 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.504421+0000", 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4329999999999998e-06 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506069+0000", 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016486770000000001 2024-04-18T21:51:49.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506101+0000", 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1415999999999999e-05 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506116+0000", 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5644000000000001e-05 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506241+0000", 2024-04-18T21:51:49.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000124549 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506448+0000", 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000207538 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.508330+0000", 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001881771 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.606434+0000", 2024-04-18T21:51:49.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.098103454000000007 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.802482+0000", 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19604839700000001 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.802604+0000", 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121915 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.581 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:125 3.14s0 3:2c76b789:::smithi09731216-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.506152+0000", 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.98874498499999997, 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29680660599999997, 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.582 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.583 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 125 2024-04-18T21:51:49.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.583 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506152+0000", 2024-04-18T21:51:49.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506152+0000", 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506154+0000", 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.813e-06 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506176+0000", 2024-04-18T21:51:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.243e-05 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506196+0000", 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9267e-05 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506212+0000", 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6773e-05 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.510086+0000", 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038735390000000001 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.510253+0000", 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016756900000000001 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511004+0000", 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00075091299999999997 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.606435+0000", 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095431019000000006 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.802798+0000", 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.196362174 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.802959+0000", 2024-04-18T21:51:49.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016110899999999999 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:126 3.14s0 3:2c76b789:::smithi09731216-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.98863988199999997, 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50118609800000002, 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.588 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 126 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:51:49.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506258+0000", 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506267+0000", 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7159999999999992e-06 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506287+0000", 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9542999999999999e-05 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506301+0000", 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4572999999999999e-05 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511483+0000", 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051813989999999997 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511616+0000", 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013376499999999999 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803750+0000", 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29213323699999999 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803998+0000", 2024-04-18T21:51:49.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024861799999999998 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.007443+0000", 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20344494699999999 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.593 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.6s4 32/31 ECSubWrite(tid=11, reqid=client.4509.0:107, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.006334+0000", 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.48856286300000001, 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30172597499999998, 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006334+0000", 2024-04-18T21:51:49.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006334+0000", 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006335+0000", 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3000000000000001e-07 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006370+0000", 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5309999999999999e-05 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006392+0000", 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1860000000000001e-05 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006417+0000", 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4765000000000001e-05 2024-04-18T21:51:49.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206650+0000", 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20023291400000001 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206666+0000", 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6364000000000001e-05 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.308030+0000", 2024-04-18T21:51:49.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10136403099999999 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.308060+0000", 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0201000000000001e-05 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.598 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.16s4 32/31 ECSubWrite(tid=12, reqid=client.4509.0:109, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.48845496500000002, 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40193662400000002, 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:51:49.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:51:49.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006443+0000", 2024-04-18T21:51:49.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999998e-07 2024-04-18T21:51:49.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006668+0000", 2024-04-18T21:51:49.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000224878 2024-04-18T21:51:49.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006685+0000", 2024-04-18T21:51:49.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6730000000000001e-05 2024-04-18T21:51:49.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006703+0000", 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8504000000000001e-05 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206788+0000", 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200085349 2024-04-18T21:51:49.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206807+0000", 2024-04-18T21:51:49.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.827e-05 2024-04-18T21:51:49.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408321+0000", 2024-04-18T21:51:49.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201514675 2024-04-18T21:51:49.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408379+0000", 2024-04-18T21:51:49.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7414999999999998e-05 2024-04-18T21:51:49.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.605 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.18s1 32/31 ECSubWrite(tid=15, reqid=client.4509.0:137, at_version=32'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.093921+0000", 2024-04-18T21:51:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.40097649000000002, 2024-04-18T21:51:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.22987353799999999, 2024-04-18T21:51:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.607 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.607 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:49.607 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.093921+0000", 2024-04-18T21:51:49.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.093921+0000", 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.093921+0000", 2024-04-18T21:51:49.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9999999999999998e-07 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.093980+0000", 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8584e-05 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.094001+0000", 2024-04-18T21:51:49.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1002e-05 2024-04-18T21:51:49.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.094015+0000", 2024-04-18T21:51:49.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4195e-05 2024-04-18T21:51:49.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.322975+0000", 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.22896054499999999 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.322988+0000", 2024-04-18T21:51:49.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2690999999999999e-05 2024-04-18T21:51:49.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:49.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.323764+0000", 2024-04-18T21:51:49.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000775539 2024-04-18T21:51:49.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:49.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.323794+0000", 2024-04-18T21:51:49.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0582e-05 2024-04-18T21:51:49.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.613 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.613 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.613 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:49.615 INFO:tasks.rados.rados.0.smithi097.stdout:37: finishing write tid 1 to smithi09731216-37 2024-04-18T21:51:49.615 INFO:tasks.rados.rados.0.smithi097.stdout:36: finishing write tid 3 to smithi09731216-36 2024-04-18T21:51:49.615 INFO:tasks.rados.rados.0.smithi097.stdout:36: oid 36 updating version 0 to 3 2024-04-18T21:51:49.615 INFO:tasks.rados.rados.0.smithi097.stdout:36: oid 36 version 3 is already newer than 1 2024-04-18T21:51:49.615 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 36 v 3 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-18T21:51:49.616 INFO:tasks.rados.rados.0.smithi097.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-18T21:51:49.616 INFO:tasks.rados.rados.0.smithi097.stdout:36: done (15 left) 2024-04-18T21:51:49.616 INFO:tasks.rados.rados.0.smithi097.stdout:49: write initial oid 49 2024-04-18T21:51:49.616 INFO:tasks.rados.rados.0.smithi097.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-04-18T21:51:49.616 INFO:tasks.rados.rados.0.smithi097.stdout:49: writing smithi09731216-49 from 0 to 442368 tid 1 2024-04-18T21:51:49.616 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:51:49.617 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:51:49.617 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:51:49.618 INFO:tasks.rados.rados.0.smithi097.stdout:49: writing smithi09731216-49 from 442368 to 819200 tid 2 2024-04-18T21:51:49.618 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:49.618 INFO:tasks.rados.rados.0.smithi097.stdout:43: finishing write tid 2 to smithi09731216-43 2024-04-18T21:51:49.618 INFO:tasks.rados.rados.0.smithi097.stdout:25: finishing write tid 3 to smithi09731216-25 2024-04-18T21:51:49.618 INFO:tasks.rados.rados.0.smithi097.stdout:25: oid 25 updating version 0 to 11 2024-04-18T21:51:49.618 INFO:tasks.rados.rados.0.smithi097.stdout:25: oid 25 version 11 is already newer than 10 2024-04-18T21:51:49.618 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 25 v 11 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-18T21:51:49.618 INFO:tasks.rados.rados.0.smithi097.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout:37: finishing write tid 2 to smithi09731216-37 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout:44: finishing write tid 1 to smithi09731216-44 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout:46: finishing write tid 1 to smithi09731216-46 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout:25: done (15 left) 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout:50: write initial oid 50 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout:50: writing smithi09731216-50 from 0 to 458752 tid 1 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout:50: writing smithi09731216-50 from 458752 to 835584 tid 2 2024-04-18T21:51:49.619 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:49.619 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:51:49.620 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:51:49.620 INFO:tasks.rados.rados.0.smithi097.stdout:45: finishing write tid 3 to smithi09731216-45 2024-04-18T21:51:49.620 INFO:tasks.rados.rados.0.smithi097.stdout:45: oid 45 updating version 0 to 10 2024-04-18T21:51:49.620 INFO:tasks.rados.rados.0.smithi097.stdout:45: oid 45 version 10 is already newer than 9 2024-04-18T21:51:49.620 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 45 v 10 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-18T21:51:49.620 INFO:tasks.rados.rados.0.smithi097.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-18T21:51:49.621 INFO:tasks.rados.rados.0.smithi097.stdout:45: done (15 left) 2024-04-18T21:51:49.621 INFO:tasks.rados.rados.0.smithi097.stdout:51: copy_from oid 23 from oid 11 current snap is 0 2024-04-18T21:51:49.621 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:49.621 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:49.611+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:49.622 INFO:tasks.rados.rados.0.smithi097.stdout:44: finishing write tid 2 to smithi09731216-44 2024-04-18T21:51:49.622 INFO:tasks.rados.rados.0.smithi097.stdout:15: finishing write tid 3 to smithi09731216-15 2024-04-18T21:51:49.622 INFO:tasks.rados.rados.0.smithi097.stdout:15: oid 15 updating version 0 to 5 2024-04-18T21:51:49.622 INFO:tasks.rados.rados.0.smithi097.stdout:15: oid 15 updating version 5 to 6 2024-04-18T21:51:49.622 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-18T21:51:49.622 INFO:tasks.rados.rados.0.smithi097.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-18T21:51:49.622 INFO:tasks.rados.rados.0.smithi097.stdout:15: done (15 left) 2024-04-18T21:51:49.623 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 28 current snap is 0 2024-04-18T21:51:49.623 INFO:tasks.rados.rados.0.smithi097.stdout:52: seq_num 50 ranges {458752=475136,933888=376832} 2024-04-18T21:51:49.632 INFO:tasks.rados.rados.0.smithi097.stdout:52: writing smithi09731216-28 from 458752 to 933888 tid 1 2024-04-18T21:51:49.638 INFO:tasks.rados.rados.0.smithi097.stdout:52: writing smithi09731216-28 from 933888 to 1310720 tid 2 2024-04-18T21:51:49.638 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:46: finishing write tid 2 to smithi09731216-46 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:17: finishing write tid 3 to smithi09731216-17 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:17: oid 17 updating version 0 to 8 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:17: oid 17 version 8 is already newer than 7 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 17 v 8 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:31: finishing write tid 3 to smithi09731216-31 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:31: oid 31 updating version 0 to 13 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:31: oid 31 version 13 is already newer than 12 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 31 v 13 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:37: finishing write tid 3 to smithi09731216-37 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:37: oid 37 updating version 0 to 4 2024-04-18T21:51:49.639 INFO:tasks.rados.rados.0.smithi097.stdout:37: oid 37 version 4 is already newer than 3 2024-04-18T21:51:49.640 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 37 v 4 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-18T21:51:49.640 INFO:tasks.rados.rados.0.smithi097.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-18T21:51:49.640 INFO:tasks.rados.rados.0.smithi097.stdout:17: done (15 left) 2024-04-18T21:51:49.640 INFO:tasks.rados.rados.0.smithi097.stdout:31: done (14 left) 2024-04-18T21:51:49.640 INFO:tasks.rados.rados.0.smithi097.stdout:37: done (13 left) 2024-04-18T21:51:49.640 INFO:tasks.rados.rados.0.smithi097.stdout:53: rmattr oid 2 current snap is 0 2024-04-18T21:51:49.640 INFO:tasks.rados.rados.0.smithi097.stdout:53: done (13 left) 2024-04-18T21:51:49.640 INFO:tasks.rados.rados.0.smithi097.stdout:54: delete oid 10 current snap is 0 2024-04-18T21:51:49.664 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:51:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:51:49.704 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:51:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:51:49.707 INFO:tasks.rados.rados.0.smithi097.stdout:47: finishing write tid 1 to smithi09731216-47 2024-04-18T21:51:49.707 INFO:tasks.rados.rados.0.smithi097.stdout:47: finishing write tid 2 to smithi09731216-47 2024-04-18T21:51:49.715 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:49.711+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:49.723 INFO:tasks.rados.rados.0.smithi097.stdout:20: finishing write tid 2 to smithi09731216-20 2024-04-18T21:51:49.725 INFO:tasks.rados.rados.0.smithi097.stdout:22: finishing write tid 1 to smithi09731216-22 2024-04-18T21:51:49.738 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:51:49.739 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:51:49.768 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:51:49.768 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:51:49.791 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:49.791 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:49.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.791 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.11s0 33/31 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.723941+0000", 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.070250355, 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.070286892000000004, 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.723941+0000", 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.723941+0000", 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.723941+0000", 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.723941+0000", 2024-04-18T21:51:49.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.723962+0000", 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0914e-05 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.723969+0000", 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8129999999999998e-06 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.794 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.1s0 33/31 ECSubRead(tid=24, to_read={3:809f338e:::smithi09731216-11:head=0,45056,0}, subchunks={3:809f338e:::smithi09731216-11:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.726198+0000", 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.067993793999999996, 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.068129141000000004, 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726198+0000", 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726198+0000", 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726198+0000", 2024-04-18T21:51:49.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726198+0000", 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726221+0000", 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2700999999999998e-05 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726232+0000", 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0885e-05 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.1s0 33/31 ECSubReadReply(tid=23, attrs_read=0))", 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.726507+0000", 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.067684889999999998, 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.067892757999999997, 2024-04-18T21:51:49.798 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726507+0000", 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726507+0000", 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726507+0000", 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726507+0000", 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726522+0000", 2024-04-18T21:51:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5551000000000001e-05 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.726528+0000", 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5799999999999999e-06 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.1s0 33/31 ECSubReadReply(tid=23, attrs_read=0))", 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.727189+0000", 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.067002928000000003, 2024-04-18T21:51:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.067281709999999995, 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727189+0000", 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727189+0000", 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727190+0000", 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6700000000000002e-07 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727202+0000", 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2536e-05 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727215+0000", 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2155e-05 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727220+0000", 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7409999999999998e-06 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.1s0 33/31 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.727707+0000", 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.066484409999999994, 2024-04-18T21:51:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.066834266000000003, 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727707+0000", 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727707+0000", 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727708+0000", 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5700000000000001e-07 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727736+0000", 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7883000000000001e-05 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727750+0000", 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4334e-05 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.727761+0000", 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0478e-05 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.1s0,GRANT,e:33,from:8(7),reservation_nonce:7)", 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.743647+0000", 2024-04-18T21:51:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.050544814, 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050968391000000002, 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.743647+0000", 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.743647+0000", 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.743648+0000", 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.031e-06 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.743652+0000", 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1819999999999999e-06 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.743667+0000", 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5046e-05 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.743675+0000", 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5560000000000002e-06 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:129 3.11s0 3:8fad7925:::smithi09731216-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.815379+0000", 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.97881314500000005, 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97932342699999997, 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 129 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815379+0000", 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815379+0000", 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815379+0000", 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0299999999999999e-07 2024-04-18T21:51:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815383+0000", 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3840000000000001e-06 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815391+0000", 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2279999999999998e-06 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815397+0000", 2024-04-18T21:51:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2210000000000002e-06 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904554+0000", 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.089157219999999995 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904666+0000", 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011152199999999999 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521861+0000", 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61719571299999998 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.522071+0000", 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000209287 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:132 3.1s0 3:87c089d9:::smithi09731216-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911941+0000", 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.88225076000000002, 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88289356600000002, 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 132 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911941+0000", 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911941+0000", 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911942+0000", 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9699999999999996e-07 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911945+0000", 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4920000000000002e-06 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911952+0000", 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0339999999999999e-06 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911957+0000", 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1000000000000003e-06 2024-04-18T21:51:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.725296+0000", 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.81333898100000002 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.725427+0000", 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013048500000000001 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(osd.3.19:1 3.1s0 3:809f338e:::smithi09731216-11:head [assert-version v4,copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.625624+0000", 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.168567366, 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16931903000000001, 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.625624+0000", 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.625624+0000", 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.625626+0000", 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9099999999999999e-06 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.625638+0000", 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1221e-05 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.625663+0000", 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5653e-05 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.625683+0000", 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9758999999999998e-05 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.725753+0000", 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100069859 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.725946+0000", 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019326099999999999 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:160 3.11s0 3.1ed60f51 (undecoded) ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e33)", 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.643155+0000", 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.15103635000000001, 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.151883554, 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.643155+0000", 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.643155+0000", 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.643158+0000", 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1390000000000002e-06 2024-04-18T21:51:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.643167+0000", 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7850000000000003e-06 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.643194+0000", 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6559000000000001e-05 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.743312+0000", 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100118149 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.10s3 33/31 ECSubWrite(tid=18, reqid=client.4509.0:140, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.712399+0000", 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.081792338000000006, 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.082727023999999996, 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.712399+0000", 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.712399+0000", 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.712401+0000", 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2470000000000001e-06 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.712414+0000", 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3715e-05 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.712438+0000", 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3163000000000002e-05 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.712453+0000", 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5031000000000001e-05 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=18, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.722281+0000", 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.071910585999999999, 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.072919558999999995, 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.722281+0000", 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.722281+0000", 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.722282+0000", 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1600000000000002e-07 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.722285+0000", 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4489999999999999e-06 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.722293+0000", 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1270000000000003e-06 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:51:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 12 2024-04-18T21:51:49.833 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:49.834 INFO:tasks.rados.rados.0.smithi097.stdout:50: finishing write tid 1 to smithi09731216-50 2024-04-18T21:51:49.834 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:51:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:51:49.835 INFO:tasks.rados.rados.0.smithi097.stdout:46: finishing write tid 3 to smithi09731216-46 2024-04-18T21:51:49.835 INFO:tasks.rados.rados.0.smithi097.stdout:46: oid 46 updating version 0 to 3 2024-04-18T21:51:49.835 INFO:tasks.rados.rados.0.smithi097.stdout:46: oid 46 updating version 3 to 4 2024-04-18T21:51:49.835 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 46 v 4 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-18T21:51:49.835 INFO:tasks.rados.rados.0.smithi097.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-18T21:51:49.836 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:49.831+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:49.836 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:51:49.911 INFO:tasks.rados.rados.0.smithi097.stdout:47: finishing write tid 3 to smithi09731216-47 2024-04-18T21:51:49.911 INFO:tasks.rados.rados.0.smithi097.stdout:47: oid 47 updating version 0 to 8 2024-04-18T21:51:49.911 INFO:tasks.rados.rados.0.smithi097.stdout:47: oid 47 version 8 is already newer than 7 2024-04-18T21:51:49.911 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 47 v 8 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-18T21:51:49.912 INFO:tasks.rados.rados.0.smithi097.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-18T21:51:49.925 INFO:tasks.rados.rados.0.smithi097.stdout:22: finishing write tid 2 to smithi09731216-22 2024-04-18T21:51:49.926 INFO:tasks.rados.rados.0.smithi097.stdout:49: finishing write tid 1 to smithi09731216-49 2024-04-18T21:51:49.935 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:49.936+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:49.950 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:51:49.950 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:51:49.950 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-04-18T21:51:49.950 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-18T21:51:49.950 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-18T21:51:49.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok config set filestore_inject_stall 3 2024-04-18T21:51:49.960 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:49.960 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:49.960 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:51:49.960 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:51:49.960 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:49.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:51:50.026 INFO:tasks.rados.rados.0.smithi097.stdout:49: finishing write tid 2 to smithi09731216-49 2024-04-18T21:51:50.027 INFO:tasks.rados.rados.0.smithi097.stdout:49: finishing write tid 3 to smithi09731216-49 2024-04-18T21:51:50.027 INFO:tasks.rados.rados.0.smithi097.stdout:20: finishing write tid 3 to smithi09731216-20 2024-04-18T21:51:50.027 INFO:tasks.rados.rados.0.smithi097.stdout:20: oid 20 updating version 0 to 9 2024-04-18T21:51:50.027 INFO:tasks.rados.rados.0.smithi097.stdout:20: oid 20 updating version 9 to 10 2024-04-18T21:51:50.027 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 20 v 10 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-18T21:51:50.028 INFO:tasks.rados.rados.0.smithi097.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-18T21:51:50.029 INFO:tasks.rados.rados.0.smithi097.stdout:22: finishing write tid 3 to smithi09731216-22 2024-04-18T21:51:50.029 INFO:tasks.rados.rados.0.smithi097.stdout:22: oid 22 updating version 0 to 11 2024-04-18T21:51:50.029 INFO:tasks.rados.rados.0.smithi097.stdout:22: oid 22 updating version 11 to 12 2024-04-18T21:51:50.029 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 22 v 12 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-18T21:51:50.029 INFO:tasks.rados.rados.0.smithi097.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-18T21:51:50.036 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:50.031+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:50.053 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:51:50.053 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:51:50.085 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:50.085 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-18T21:51:50.085 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:50.101 INFO:tasks.rados.rados.0.smithi097.stdout:50: finishing write tid 2 to smithi09731216-50 2024-04-18T21:51:50.102 INFO:tasks.rados.rados.0.smithi097.stdout:50: finishing write tid 3 to smithi09731216-50 2024-04-18T21:51:50.136 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:50.135+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:50.139 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:50.140 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:50.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:51:50.224 INFO:tasks.rados.rados.0.smithi097.stdout:43: finishing write tid 3 to smithi09731216-43 2024-04-18T21:51:50.224 INFO:tasks.rados.rados.0.smithi097.stdout:43: oid 43 updating version 0 to 5 2024-04-18T21:51:50.224 INFO:tasks.rados.rados.0.smithi097.stdout:43: oid 43 updating version 5 to 6 2024-04-18T21:51:50.224 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-18T21:51:50.225 INFO:tasks.rados.rados.0.smithi097.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-18T21:51:50.225 INFO:tasks.rados.rados.0.smithi097.stdout:44: finishing write tid 3 to smithi09731216-44 2024-04-18T21:51:50.225 INFO:tasks.rados.rados.0.smithi097.stdout:44: oid 44 updating version 0 to 6 2024-04-18T21:51:50.225 INFO:tasks.rados.rados.0.smithi097.stdout:44: oid 44 version 6 is already newer than 5 2024-04-18T21:51:50.225 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-18T21:51:50.225 INFO:tasks.rados.rados.0.smithi097.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-18T21:51:50.228 INFO:tasks.rados.rados.0.smithi097.stdout:51: finishing copy_from racing read to smithi09731216-23 2024-04-18T21:51:50.236 INFO:tasks.rados.rados.0.smithi097.stdout:52: finishing write tid 1 to smithi09731216-28 2024-04-18T21:51:50.236 INFO:tasks.rados.rados.0.smithi097.stdout:52: finishing write tid 2 to smithi09731216-28 2024-04-18T21:51:50.236 INFO:tasks.rados.rados.0.smithi097.stdout:52: finishing write tid 3 to smithi09731216-28 2024-04-18T21:51:50.237 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:50.235+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:20: done (13 left) 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:22: done (12 left) 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:43: done (11 left) 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:44: done (10 left) 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:46: done (9 left) 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:47: done (8 left) 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:54: done (7 left) 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:55: read oid 31 snap -1 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:55: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-18T21:51:50.240 INFO:tasks.rados.rados.0.smithi097.stdout:56: rollback oid 34 current snap is 0 2024-04-18T21:51:50.241 INFO:tasks.rados.rados.0.smithi097.stdout:56: done (8 left) 2024-04-18T21:51:50.241 INFO:tasks.rados.rados.0.smithi097.stdout:57: read oid 30 snap -1 2024-04-18T21:51:50.241 INFO:tasks.rados.rados.0.smithi097.stdout:57: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-18T21:51:50.241 INFO:tasks.rados.rados.0.smithi097.stdout:58: copy_from oid 17 from oid 42 current snap is 0 2024-04-18T21:51:50.241 INFO:tasks.rados.rados.0.smithi097.stdout:59: rollback oid 22 current snap is 0 2024-04-18T21:51:50.241 INFO:tasks.rados.rados.0.smithi097.stdout:59: done (10 left) 2024-04-18T21:51:50.241 INFO:tasks.rados.rados.0.smithi097.stdout:60: copy_from oid 18 from oid 15 current snap is 0 2024-04-18T21:51:50.241 INFO:tasks.rados.rados.0.smithi097.stdout:61: delete oid 16 current snap is 0 2024-04-18T21:51:50.290 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:50.290 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:50.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:51:50.305 INFO:tasks.rados.rados.0.smithi097.stdout:49: finishing write tid 4 to smithi09731216-49 2024-04-18T21:51:50.305 INFO:tasks.rados.rados.0.smithi097.stdout:49: oid 49 updating version 0 to 5 2024-04-18T21:51:50.305 INFO:tasks.rados.rados.0.smithi097.stdout:49: oid 49 version 5 is already newer than 3 2024-04-18T21:51:50.305 INFO:tasks.rados.rados.0.smithi097.stdout:49: oid 49 version 5 is already newer than 4 2024-04-18T21:51:50.305 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 49 v 5 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-18T21:51:50.305 INFO:tasks.rados.rados.0.smithi097.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-18T21:51:50.337 INFO:tasks.rados.rados.0.smithi097.stdout:29: finishing write tid 1 to smithi09731216-29 2024-04-18T21:51:50.338 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:50.335+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:50.340 INFO:tasks.rados.rados.0.smithi097.stdout:29: finishing write tid 2 to smithi09731216-29 2024-04-18T21:51:50.407 INFO:tasks.rados.rados.0.smithi097.stdout:50: finishing write tid 4 to smithi09731216-50 2024-04-18T21:51:50.407 INFO:tasks.rados.rados.0.smithi097.stdout:50: oid 50 updating version 0 to 6 2024-04-18T21:51:50.407 INFO:tasks.rados.rados.0.smithi097.stdout:50: oid 50 updating version 6 to 7 2024-04-18T21:51:50.407 INFO:tasks.rados.rados.0.smithi097.stdout:50: oid 50 version 7 is already newer than 5 2024-04-18T21:51:50.407 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 50 v 7 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-18T21:51:50.407 INFO:tasks.rados.rados.0.smithi097.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-18T21:51:50.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:50.435+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:50.441 INFO:tasks.rados.rados.0.smithi097.stdout:51: finishing copy_from to smithi09731216-23 2024-04-18T21:51:50.442 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 23 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-18T21:51:50.458 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:50.458 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:50.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:51:50.529 INFO:tasks.rados.rados.0.smithi097.stdout:49: done (11 left) 2024-04-18T21:51:50.530 INFO:tasks.rados.rados.0.smithi097.stdout:50: done (10 left) 2024-04-18T21:51:50.530 INFO:tasks.rados.rados.0.smithi097.stdout:51: done (9 left) 2024-04-18T21:51:50.530 INFO:tasks.rados.rados.0.smithi097.stdout:61: done (8 left) 2024-04-18T21:51:50.530 INFO:tasks.rados.rados.0.smithi097.stdout:62: read oid 32 snap -1 2024-04-18T21:51:50.530 INFO:tasks.rados.rados.0.smithi097.stdout:62: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:51:50.530 INFO:tasks.rados.rados.0.smithi097.stdout:63: snap_create 2024-04-18T21:51:50.534 INFO:tasks.rados.rados.0.smithi097.stdout:38: finishing write tid 1 to smithi09731216-38 2024-04-18T21:51:50.534 INFO:tasks.rados.rados.0.smithi097.stdout:48: finishing write tid 1 to smithi09731216-48 2024-04-18T21:51:50.538 INFO:tasks.rados.rados.0.smithi097.stdout:52: finishing write tid 4 to smithi09731216-28 2024-04-18T21:51:50.538 INFO:tasks.rados.rados.0.smithi097.stdout:52: oid 28 updating version 0 to 7 2024-04-18T21:51:50.538 INFO:tasks.rados.rados.0.smithi097.stdout:52: oid 28 updating version 7 to 9 2024-04-18T21:51:50.538 INFO:tasks.rados.rados.0.smithi097.stdout:52: oid 28 version 9 is already newer than 8 2024-04-18T21:51:50.538 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 28 v 9 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-18T21:51:50.538 INFO:tasks.rados.rados.0.smithi097.stdout:52: left oid 28 (ObjNum 50 snap 0 seq_num 50) 2024-04-18T21:51:50.539 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:50.540+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:50.635 INFO:tasks.rados.rados.0.smithi097.stdout:57: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-18T21:51:50.637 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:50.638 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:50.640 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:50.635+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:50.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:51:50.673 INFO:tasks.rados.rados.0.smithi097.stdout:48: finishing write tid 2 to smithi09731216-48 2024-04-18T21:51:50.673 INFO:tasks.rados.rados.0.smithi097.stdout:38: finishing write tid 2 to smithi09731216-38 2024-04-18T21:51:50.673 INFO:tasks.rados.rados.0.smithi097.stdout:48: finishing write tid 3 to smithi09731216-48 2024-04-18T21:51:50.741 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:50.739+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:50.748 INFO:tasks.rados.rados.0.smithi097.stdout:29: finishing write tid 3 to smithi09731216-29 2024-04-18T21:51:50.749 INFO:tasks.rados.rados.0.smithi097.stdout:29: oid 29 updating version 0 to 14 2024-04-18T21:51:50.749 INFO:tasks.rados.rados.0.smithi097.stdout:29: oid 29 version 14 is already newer than 13 2024-04-18T21:51:50.749 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 29 v 14 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-18T21:51:50.749 INFO:tasks.rados.rados.0.smithi097.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-18T21:51:50.794 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:50.795 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:50.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:51:50.836 INFO:tasks.rados.rados.0.smithi097.stdout:62: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:51:50.841 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:50.839+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:50.928 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:50.928 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:50.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:51:50.943 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:50.944+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:50.991 INFO:tasks.rados.rados.0.smithi097.stdout:29: done (9 left) 2024-04-18T21:51:50.991 INFO:tasks.rados.rados.0.smithi097.stdout:52: done (8 left) 2024-04-18T21:51:50.991 INFO:tasks.rados.rados.0.smithi097.stdout:57: done (7 left) 2024-04-18T21:51:50.992 INFO:tasks.rados.rados.0.smithi097.stdout:62: done (6 left) 2024-04-18T21:51:50.992 INFO:tasks.rados.rados.0.smithi097.stdout:63: done (5 left) 2024-04-18T21:51:50.992 INFO:tasks.rados.rados.0.smithi097.stdout:64: copy_from oid 35 from oid 2 current snap is 1 2024-04-18T21:51:50.992 INFO:tasks.rados.rados.0.smithi097.stdout:65: read oid 14 snap -1 2024-04-18T21:51:50.992 INFO:tasks.rados.rados.0.smithi097.stdout:65: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-18T21:51:50.992 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 9 current snap is 1 2024-04-18T21:51:50.992 INFO:tasks.rados.rados.0.smithi097.stdout:66: seq_num 53 ranges {147456=524288,671744=376832} 2024-04-18T21:51:50.996 INFO:tasks.rados.rados.0.smithi097.stdout:66: writing smithi09731216-9 from 147456 to 671744 tid 1 2024-04-18T21:51:50.999 INFO:tasks.rados.rados.0.smithi097.stdout:66: writing smithi09731216-9 from 671744 to 1048576 tid 2 2024-04-18T21:51:51.000 INFO:tasks.rados.rados.0.smithi097.stdout:67: rollback oid 46 current snap is 1 2024-04-18T21:51:51.000 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 46 to 0 2024-04-18T21:51:51.000 INFO:tasks.rados.rados.0.smithi097.stdout:68: setattr oid 6 current snap is 1 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:69: copy_from oid 8 from oid 5 current snap is 1 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:70: read oid 28 snap 0 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:70: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:71: copy_from oid 3 from oid 23 current snap is 1 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:72: rollback oid 10 current snap is 1 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 10 to 0 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:73: read oid 16 snap 0 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:73: expect deleted 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:74: rmattr oid 25 current snap is 1 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:74: done (15 left) 2024-04-18T21:51:51.001 INFO:tasks.rados.rados.0.smithi097.stdout:75: copy_from oid 44 from oid 33 current snap is 1 2024-04-18T21:51:51.002 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:51.043 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:51.039+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:51.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-18T21:51:51.101 INFO:tasks.rados.rados.0.smithi097.stdout:55: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-18T21:51:51.116 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:51.116 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:51.122 INFO:tasks.rados.rados.0.smithi097.stdout:55: done (15 left) 2024-04-18T21:51:51.122 INFO:tasks.rados.rados.0.smithi097.stdout:73: done (14 left) 2024-04-18T21:51:51.122 INFO:tasks.rados.rados.0.smithi097.stdout:76: read oid 31 snap -1 2024-04-18T21:51:51.122 INFO:tasks.rados.rados.0.smithi097.stdout:76: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-18T21:51:51.122 INFO:tasks.rados.rados.0.smithi097.stdout:77: copy_from oid 25 from oid 49 current snap is 1 2024-04-18T21:51:51.123 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:51.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:51:51.177 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:51.175+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:51.245 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:51.239+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:51.299 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:51.300 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:51.313 INFO:tasks.rados.rados.0.smithi097.stdout:70: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-18T21:51:51.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:51:51.345 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:51.348+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:51.392 INFO:tasks.rados.rados.0.smithi097.stdout:70: done (15 left) 2024-04-18T21:51:51.392 INFO:tasks.rados.rados.0.smithi097.stdout:78: read oid 30 snap -1 2024-04-18T21:51:51.393 INFO:tasks.rados.rados.0.smithi097.stdout:78: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-18T21:51:51.393 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:51.412 INFO:tasks.rados.rados.0.smithi097.stdout:72: finishing rollback tid 1 to smithi09731216-10 2024-04-18T21:51:51.412 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 10 v 0 (ObjNum 49 snap 0 seq_num 815817876) dirty dne 2024-04-18T21:51:51.412 INFO:tasks.rados.rados.0.smithi097.stdout:72: done (15 left) 2024-04-18T21:51:51.412 INFO:tasks.rados.rados.0.smithi097.stdout:79: setattr oid 27 current snap is 1 2024-04-18T21:51:51.412 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 16 2024-04-18T21:51:51.447 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:51.443+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:51.457 INFO:tasks.rados.rados.0.smithi097.stdout:38: finishing write tid 3 to smithi09731216-38 2024-04-18T21:51:51.457 INFO:tasks.rados.rados.0.smithi097.stdout:38: oid 38 updating version 0 to 16 2024-04-18T21:51:51.457 INFO:tasks.rados.rados.0.smithi097.stdout:38: oid 38 version 16 is already newer than 15 2024-04-18T21:51:51.457 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 38 v 16 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-18T21:51:51.457 INFO:tasks.rados.rados.0.smithi097.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-18T21:51:51.457 INFO:tasks.rados.rados.0.smithi097.stdout:38: done (15 left) 2024-04-18T21:51:51.457 INFO:tasks.rados.rados.0.smithi097.stdout:80: snap_create 2024-04-18T21:51:51.478 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:51.478 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:51.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:51:51.548 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:51.543+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:51.557 INFO:tasks.rados.rados.0.smithi097.stdout:48: finishing write tid 4 to smithi09731216-48 2024-04-18T21:51:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:48: oid 48 updating version 0 to 15 2024-04-18T21:51:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:48: oid 48 version 15 is already newer than 14 2024-04-18T21:51:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:48: oid 48 updating version 15 to 16 2024-04-18T21:51:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 48 v 16 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-18T21:51:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-18T21:51:51.561 INFO:tasks.rados.rados.0.smithi097.stdout:67: finishing rollback tid 0 to smithi09731216-46 2024-04-18T21:51:51.561 INFO:tasks.rados.rados.0.smithi097.stdout:67: finishing rollback tid 1 to smithi09731216-46 2024-04-18T21:51:51.561 INFO:tasks.rados.rados.0.smithi097.stdout:67: finishing rollback tid 2 to smithi09731216-46 2024-04-18T21:51:51.562 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 46 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-18T21:51:51.567 INFO:tasks.rados.rados.0.smithi097.stdout:66: finishing write tid 1 to smithi09731216-9 2024-04-18T21:51:51.567 INFO:tasks.rados.rados.0.smithi097.stdout:66: finishing write tid 2 to smithi09731216-9 2024-04-18T21:51:51.568 INFO:tasks.rados.rados.0.smithi097.stdout:66: finishing write tid 3 to smithi09731216-9 2024-04-18T21:51:51.571 INFO:tasks.rados.rados.0.smithi097.stdout:64: finishing copy_from racing read to smithi09731216-35 2024-04-18T21:51:51.574 INFO:tasks.rados.rados.0.smithi097.stdout:60: finishing copy_from racing read to smithi09731216-18 2024-04-18T21:51:51.578 INFO:tasks.rados.rados.0.smithi097.stdout:69: finishing copy_from racing read to smithi09731216-8 2024-04-18T21:51:51.649 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:51.647+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:51.658 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 6 v 18 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-18T21:51:51.659 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:51.659 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:51.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:51:51.679 INFO:tasks.rados.rados.0.smithi097.stdout:75: finishing copy_from racing read to smithi09731216-44 2024-04-18T21:51:51.749 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:51.747+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:51.806 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:51.806 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:51.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:51:51.847 INFO:tasks.rados.rados.0.smithi097.stdout:78: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-18T21:51:51.850 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:51.847+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:51.867 INFO:tasks.rados.rados.0.smithi097.stdout:58: finishing copy_from racing read to smithi09731216-17 2024-04-18T21:51:51.870 INFO:tasks.rados.rados.0.smithi097.stdout:65: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-18T21:51:51.878 INFO:tasks.rados.rados.0.smithi097.stdout:66: finishing write tid 4 to smithi09731216-9 2024-04-18T21:51:51.878 INFO:tasks.rados.rados.0.smithi097.stdout:66: oid 9 updating version 0 to 8 2024-04-18T21:51:51.878 INFO:tasks.rados.rados.0.smithi097.stdout:66: oid 9 updating version 8 to 9 2024-04-18T21:51:51.878 INFO:tasks.rados.rados.0.smithi097.stdout:66: oid 9 updating version 9 to 10 2024-04-18T21:51:51.879 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 9 v 10 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-04-18T21:51:51.879 INFO:tasks.rados.rados.0.smithi097.stdout:66: left oid 9 (ObjNum 53 snap 1 seq_num 53) 2024-04-18T21:51:51.944 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:51.945 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:51.951 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:51.947+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:51.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:51:51.962 INFO:tasks.rados.rados.0.smithi097.stdout:58: finishing copy_from to smithi09731216-17 2024-04-18T21:51:51.962 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 17 v 4 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-18T21:51:51.970 INFO:tasks.rados.rados.0.smithi097.stdout:69: finishing copy_from to smithi09731216-8 2024-04-18T21:51:51.970 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 8 v 4 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-18T21:51:51.975 INFO:tasks.rados.rados.0.smithi097.stdout:75: finishing copy_from to smithi09731216-44 2024-04-18T21:51:51.975 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 44 v 2 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-18T21:51:51.995 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is set 2024-04-18T21:51:51.998 INFO:tasks.rados.rados.0.smithi097.stdout:48: done (15 left) 2024-04-18T21:51:51.998 INFO:tasks.rados.rados.0.smithi097.stdout:58: done (14 left) 2024-04-18T21:51:51.998 INFO:tasks.rados.rados.0.smithi097.stdout:65: done (13 left) 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:66: done (12 left) 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:67: done (11 left) 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:68: done (10 left) 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:69: done (9 left) 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:75: done (8 left) 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:78: done (7 left) 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:80: done (6 left) 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:81: rollback oid 4 current snap is 2 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 4 to 1 2024-04-18T21:51:51.999 INFO:tasks.rados.rados.0.smithi097.stdout:82: snap_create 2024-04-18T21:51:52.052 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:52.051+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:52.070 INFO:tasks.rados.rados.0.smithi097.stdout:64: finishing copy_from to smithi09731216-35 2024-04-18T21:51:52.070 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 35 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-18T21:51:52.076 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:51:52.081 INFO:tasks.rados.rados.0.smithi097.stdout:60: finishing copy_from to smithi09731216-18 2024-04-18T21:51:52.081 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 18 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-18T21:51:52.082 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 27 v 13 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-18T21:51:52.092 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:52.093 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:52.108 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:51:52.156 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:52.155+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:52.221 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:51:52.222 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:51:52.238 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:51:52.238 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:51:52.243 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:52.243 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:52.253 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:52.251+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:52.254 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:51:52.255 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:51:52.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:51:52.266 INFO:tasks.rados.rados.0.smithi097.stdout:76: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-18T21:51:52.272 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:51:52.272 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:51:52.291 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:51:52.291 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:51:52.312 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:51:52.312 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:51:52.329 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:51:52.329 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:51:52.347 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:51:52.347 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:51:52.353 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:52.351+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:52.368 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:51:52.368 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:51:52.389 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:51:52.390 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:51:52.424 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:52.424 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:52.424 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:52.425 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:52.425 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:52.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:51:52.454 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:52.451+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:52.502 INFO:tasks.rados.rados.0.smithi097.stdout:81: finishing rollback tid 0 to smithi09731216-4 2024-04-18T21:51:52.503 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:51:52.503 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:51:52.532 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:51:52.533 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:51:52.555 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:52.551+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:52.560 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:52.560 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:52.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:52.560 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:52.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3050465820000001, 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82320560300000001, 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254837+0000", 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9000000000000002e-06 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255465+0000", 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000628435 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255498+0000", 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2305999999999999e-05 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255523+0000", 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5137e-05 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078012+0000", 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82248975599999996 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078039+0000", 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7069000000000001e-05 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:52.564 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3043196769999996, 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82259326700000002, 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:51:52.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255563+0000", 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.72e-06 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255574+0000", 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0993e-05 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255598+0000", 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4329999999999999e-05 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255615+0000", 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7569000000000001e-05 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078131+0000", 2024-04-18T21:51:52.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82251584300000002 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078154+0000", 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2812000000000001e-05 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:93 3.3s0 3.fef99513 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3042317350000001, 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822627518, 2024-04-18T21:51:52.568 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 93 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255650+0000", 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2330000000000001e-06 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255657+0000", 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5000000000000002e-06 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255681+0000", 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3354e-05 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255705+0000", 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3932999999999999e-05 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078240+0000", 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822535609 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078276+0000", 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5889000000000001e-05 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.264649+0000", 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.2952311270000001, 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.406117544, 2024-04-18T21:51:52.572 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264649+0000", 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264649+0000", 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264651+0000", 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7460000000000001e-06 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264678+0000", 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6800000000000001e-05 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264708+0000", 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0303000000000001e-05 2024-04-18T21:51:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264727+0000", 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8966000000000001e-05 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.670698+0000", 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40597100000000003 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.670767+0000", 2024-04-18T21:51:52.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8728999999999999e-05 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s0 29/26 ECSubRead(tid=11, to_read={3:c022028e:::smithi09731216-15:head=0,4096,0}, subchunks={3:c022028e:::smithi09731216-15:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.264759+0000", 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.2951211479999998, 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.406330001, 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.576 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264759+0000", 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264759+0000", 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264759+0000", 2024-04-18T21:51:52.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264759+0000", 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264824+0000", 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4966000000000001e-05 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.264856+0000", 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1659000000000003e-05 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.670829+0000", 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40597346400000001 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.671089+0000", 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000259912 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.579 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.265477+0000", 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.2944035439999997, 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40571251200000003, 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265477+0000", 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265477+0000", 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265485+0000", 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.123e-06 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265511+0000", 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5622999999999998e-05 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265536+0000", 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5106999999999999e-05 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265553+0000", 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7563000000000001e-05 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.671137+0000", 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40558354499999999 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.671189+0000", 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2550999999999999e-05 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:51:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.265591+0000", 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.2942898779999998, 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50576242900000001, 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265591+0000", 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265591+0000", 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265592+0000", 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2130000000000001e-06 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265611+0000", 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8923000000000002e-05 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265625+0000", 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4763000000000001e-05 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265637+0000", 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1063999999999999e-05 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.771295+0000", 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50565803499999995 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.771353+0000", 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8430999999999997e-05 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.265672+0000", 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.2942087989999997, 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50577193600000003, 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.587 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265672+0000", 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265672+0000", 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265673+0000", 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.094e-06 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265692+0000", 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9516000000000001e-05 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265716+0000", 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.385e-05 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265733+0000", 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6419999999999999e-05 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.771403+0000", 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50567023200000005 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.771444+0000", 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0824000000000001e-05 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.265921+0000", 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.2939591540000004, 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60568457799999997, 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.591 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265921+0000", 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265921+0000", 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265922+0000", 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1170000000000001e-06 2024-04-18T21:51:52.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265938+0000", 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5423000000000001e-05 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265951+0000", 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3617e-05 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265962+0000", 2024-04-18T21:51:52.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0890000000000001e-05 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871553+0000", 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60559044799999995 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871606+0000", 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3083e-05 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.594 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s0 29/26 ECSubRead(tid=12, to_read={3:c057a400:::smithi09731216-17:head=0,4096,0}, subchunks={3:c057a400:::smithi09731216-17:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1920074620000003, 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50402778100000001, 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367901+0000", 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8042e-05 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367920+0000", 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8536999999999999e-05 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871645+0000", 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50372545800000001 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871901+0000", 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025574400000000002 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=12, attrs_read=0))", 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.368470+0000", 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1914102839999998, 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50362052899999998, 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368470+0000", 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368470+0000", 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368471+0000", 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8000000000000005e-07 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368481+0000", 2024-04-18T21:51:52.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0183000000000001e-05 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368492+0000", 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0710000000000001e-05 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.468570+0000", 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100077795 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871983+0000", 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40341337799999999 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.872091+0000", 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010768300000000001 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=12, attrs_read=0))", 2024-04-18T21:51:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.468971+0000", 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.0909090880000001, 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40321823499999998, 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.468971+0000", 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.468971+0000", 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.468973+0000", 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.024e-06 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469020+0000", 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6993000000000002e-05 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469050+0000", 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9522999999999998e-05 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469071+0000", 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0979e-05 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.872148+0000", 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40307695700000001 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.872190+0000", 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1758999999999998e-05 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=12, attrs_read=0))", 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.469884+0000", 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.0899969509999998, 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40236560799999999, 2024-04-18T21:51:52.606 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469884+0000", 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469884+0000", 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469884+0000", 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1800000000000004e-07 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469900+0000", 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5522000000000001e-05 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469915+0000", 2024-04-18T21:51:52.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5367e-05 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.469925+0000", 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6660000000000003e-06 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.872214+0000", 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.402288639 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.872249+0000", 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5496e-05 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 30/26 ECSubWriteReply(tid=13, last_complete=29'9, committed=1, applied=1))", 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.470772+0000", 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.0891083269999999, 2024-04-18T21:51:52.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40244305600000002, 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.470772+0000", 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.470772+0000", 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.470772+0000", 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0499999999999999e-07 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.470775+0000", 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3149999999999999e-06 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.470782+0000", 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5750000000000003e-06 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.570870+0000", 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100087941 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.873173+0000", 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30230304299999999 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.873215+0000", 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1876999999999999e-05 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 30/26 ECSubWriteReply(tid=13, last_complete=29'9, committed=1, applied=1))", 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.571062+0000", 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.9888182719999996, 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.402837165, 2024-04-18T21:51:52.614 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.571062+0000", 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.571062+0000", 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.571063+0000", 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3600000000000003e-07 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.571069+0000", 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7039999999999999e-06 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.571082+0000", 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3363000000000001e-05 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.571093+0000", 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.058e-05 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.973850+0000", 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.402757213 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.973899+0000", 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9568999999999999e-05 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs2 30/26 ECSubWrite(tid=11, reqid=client.4509.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.668803+0000", 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.8910773540000001, 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40783071599999998, 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668803+0000", 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668803+0000", 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668805+0000", 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.533e-06 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668819+0000", 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4188e-05 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668842+0000", 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3660999999999999e-05 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668856+0000", 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3825000000000001e-05 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.873265+0000", 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.204408969 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.873280+0000", 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.45e-05 2024-04-18T21:51:52.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.076591+0000", 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20331132399999999 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.076634+0000", 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2716000000000002e-05 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:73 3.13s0 3:c8da3643:::smithi09731216-25:head [append 0~409600,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.3631482650000004, 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.609707234, 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.623 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196734+0000", 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829999999999999e-06 2024-04-18T21:51:52.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196977+0000", 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024357200000000001 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196998+0000", 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0503e-05 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197013+0000", 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5536000000000002e-05 2024-04-18T21:51:52.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197060+0000", 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6519e-05 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197061+0000", 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2900000000000002e-07 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591261+0000", 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3942002449999999 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591644+0000", 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038315000000000002 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593905+0000", 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022609050000000001 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:52.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697736+0000", 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10383110700000001 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806236+0000", 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10850008999999999 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806439+0000", 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020309500000000001 2024-04-18T21:51:52.628 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:74 3.13s0 3:c8da3643:::smithi09731216-25:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.3628439209999996, 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7102205860000002, 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:52.629 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 74 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197038+0000", 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9000000000000005e-07 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197043+0000", 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197053+0000", 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0267e-05 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197066+0000", 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.327e-05 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197104+0000", 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8039e-05 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197105+0000", 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8400000000000004e-07 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595282+0000", 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3981771869999999 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595543+0000", 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026070899999999997 2024-04-18T21:51:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.596517+0000", 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00097469500000000003 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697956+0000", 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10143811899999999 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:52.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907075+0000", 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20911996799999999 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907257+0000", 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018164799999999999 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.13s0 3:c8a99f7f:::smithi09731216-31:head [append 0~507904,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:52.635 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.3623178579999999, 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.812483013, 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8599999999999998e-07 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197879+0000", 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000315495 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197893+0000", 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4143e-05 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197908+0000", 2024-04-18T21:51:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4932999999999999e-05 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197987+0000", 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9288000000000004e-05 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197989+0000", 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.137e-06 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.698727+0000", 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.500737242 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.699015+0000", 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028849499999999999 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.700958+0000", 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019429969999999999 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:52.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798717+0000", 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097759288999999999 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.009932+0000", 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.211214651 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010046+0000", 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113657 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.13s0 3:c8a99f7f:::smithi09731216-31:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.3619560169999998, 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8124102710000001, 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:51:52.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197925+0000", 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000005e-07 2024-04-18T21:51:52.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197931+0000", 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6799999999999998e-06 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197943+0000", 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1620000000000001e-05 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197953+0000", 2024-04-18T21:51:52.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-05 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7586000000000001e-05 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5599999999999995e-07 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702431+0000", 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.504419629 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702670+0000", 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023948700000000001 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.703977+0000", 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00130674 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798719+0000", 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094742397000000006 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010219+0000", 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21149920999999999 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010335+0000", 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011602 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.647 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:52.648 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:52.648 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:52.649 INFO:tasks.rados.rados.0.smithi097.stdout:71: finishing copy_from racing read to smithi09731216-3 2024-04-18T21:51:52.649 INFO:tasks.rados.rados.0.smithi097.stdout:77: finishing copy_from racing read to smithi09731216-25 2024-04-18T21:51:52.649 INFO:tasks.rados.rados.0.smithi097.stdout:81: finishing rollback tid 1 to smithi09731216-4 2024-04-18T21:51:52.649 INFO:tasks.rados.rados.0.smithi097.stdout:81: finishing rollback tid 2 to smithi09731216-4 2024-04-18T21:51:52.649 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 4 v 13 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-18T21:51:52.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:51:52.657 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:52.651+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:52.686 INFO:tasks.rados.rados.0.smithi097.stdout:71: finishing copy_from to smithi09731216-3 2024-04-18T21:51:52.686 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 3 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-18T21:51:52.757 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:52.755+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:52.776 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:52.776 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:52.776 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:51:52.776 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:52.788 INFO:tasks.rados.rados.0.smithi097.stdout:77: finishing copy_from to smithi09731216-25 2024-04-18T21:51:52.788 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 25 v 5 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-18T21:51:52.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:51:52.858 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:52.855+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:52.956 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:52.956 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:52.959 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:52.959+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:52.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:51:53.012 INFO:tasks.rados.rados.0.smithi097.stdout:60: done (7 left) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:64: done (6 left) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:71: done (5 left) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:76: done (4 left) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:77: done (3 left) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:79: done (2 left) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:81: done (1 left) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:82: done (0 left) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:83: rollback oid 9 current snap is 3 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 9 to 0 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:84: read oid 20 snap 1 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:84: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-18T21:51:53.013 INFO:tasks.rados.rados.0.smithi097.stdout:85: read oid 41 snap -1 2024-04-18T21:51:53.014 INFO:tasks.rados.rados.0.smithi097.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-18T21:51:53.014 INFO:tasks.rados.rados.0.smithi097.stdout:86: read oid 5 snap -1 2024-04-18T21:51:53.014 INFO:tasks.rados.rados.0.smithi097.stdout:86: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-18T21:51:53.014 INFO:tasks.rados.rados.0.smithi097.stdout:87: read oid 4 snap -1 2024-04-18T21:51:53.014 INFO:tasks.rados.rados.0.smithi097.stdout:87: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-18T21:51:53.014 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 3 current snap is 3 2024-04-18T21:51:53.014 INFO:tasks.rados.rados.0.smithi097.stdout:88: seq_num 54 ranges {180224=540672,720896=376832} 2024-04-18T21:51:53.025 INFO:tasks.rados.rados.0.smithi097.stdout:88: writing smithi09731216-3 from 180224 to 720896 tid 1 2024-04-18T21:51:53.032 INFO:tasks.rados.rados.0.smithi097.stdout:88: writing smithi09731216-3 from 720896 to 1097728 tid 2 2024-04-18T21:51:53.032 INFO:tasks.rados.rados.0.smithi097.stdout:89: read oid 27 snap -1 2024-04-18T21:51:53.032 INFO:tasks.rados.rados.0.smithi097.stdout:89: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-18T21:51:53.032 INFO:tasks.rados.rados.0.smithi097.stdout:90: snap_remove snap 2 2024-04-18T21:51:53.060 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:53.055+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:53.099 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:53.099 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:53.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:53.099 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 30/26 ECSubWriteReply(tid=18, last_complete=30'4, committed=1, applied=1))", 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.659637+0000", 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.4373947640000004, 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20044604999999999, 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659637+0000", 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659637+0000", 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659638+0000", 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9500000000000003e-07 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659642+0000", 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7120000000000002e-06 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659650+0000", 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9999999999999996e-06 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759760+0000", 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100110484 2024-04-18T21:51:53.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.860017+0000", 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100256967 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.860083+0000", 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6391999999999997e-05 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:100 3.cs0 3:348e44d8:::smithi09731216-34:head [append 0~557056,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.4370804640000001, 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50652407300000002, 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659953+0000", 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.821e-06 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660381+0000", 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042725800000000002 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660395+0000", 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.394e-05 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660404+0000", 2024-04-18T21:51:53.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6520000000000001e-06 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760564+0000", 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10015995900000001 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760699+0000", 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013526 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762105+0000", 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00140595 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965039+0000", 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20293309400000001 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166421+0000", 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201382373 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166476+0000", 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4765999999999997e-05 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:101 3.cs0 3:348e44d8:::smithi09731216-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.4366107240000003, 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60956795200000002, 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.109 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660422+0000", 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-07 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660426+0000", 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3830000000000002e-06 2024-04-18T21:51:53.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660436+0000", 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8679999999999994e-06 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660443+0000", 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2999999999999998e-06 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762872+0000", 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1024294 2024-04-18T21:51:53.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762973+0000", 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001015 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763506+0000", 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053209199999999996 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965047+0000", 2024-04-18T21:51:53.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20154100999999999 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269839+0000", 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.304792377 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269989+0000", 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015033399999999999 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.114 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:102 3.cs0 3:348e44d8:::smithi09731216-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.4365611749999996, 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.92759091000000005, 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660472+0000", 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660476+0000", 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0879999999999997e-06 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660493+0000", 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6963e-05 2024-04-18T21:51:53.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660502+0000", 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9579999999999996e-06 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763793+0000", 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10329121600000001 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:53.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763891+0000", 2024-04-18T21:51:53.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8127000000000004e-05 2024-04-18T21:51:53.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372039+0000", 2024-04-18T21:51:53.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60814838199999999 2024-04-18T21:51:53.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372189+0000", 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014991200000000001 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.588062+0000", 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.215872654 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.120 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.5s0 30/26 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.759806+0000", 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3372257210000003, 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20049813399999999, 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759806+0000", 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759806+0000", 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759808+0000", 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.164e-06 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759828+0000", 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0741999999999999e-05 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759840+0000", 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1308000000000001e-05 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759847+0000", 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4989999999999997e-06 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.960231+0000", 2024-04-18T21:51:53.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20038387799999999 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.960304+0000", 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3542999999999997e-05 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.5s0 30/26 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:51:53.124 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3371666869999999, 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30082227499999997, 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:51:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759866+0000", 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.34e-07 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759873+0000", 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.973e-06 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759879+0000", 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5440000000000004e-06 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759885+0000", 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3340000000000001e-06 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.060439+0000", 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30055447000000002 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.060688+0000", 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024862000000000002 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.4s3 30/26 ECSubWrite(tid=10, reqid=client.4509.0:97, at_version=30'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.760141+0000", 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3368911380000004, 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20499564100000001, 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.128 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760141+0000", 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760141+0000", 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760141+0000", 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.13e-07 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760254+0000", 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011241799999999999 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760263+0000", 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.217e-06 2024-04-18T21:51:53.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760269+0000", 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5740000000000003e-06 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762841+0000", 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025722560000000002 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762852+0000", 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0821e-05 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965053+0000", 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20220110499999999 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965137+0000", 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3737000000000001e-05 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.4s3 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.760281+0000", 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3367513769999997, 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20508437099999999, 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760281+0000", 2024-04-18T21:51:53.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760281+0000", 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760281+0000", 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9400000000000001e-07 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760291+0000", 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5840000000000001e-06 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760296+0000", 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2959999999999998e-06 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760306+0000", 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6339999999999996e-06 2024-04-18T21:51:53.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.863516+0000", 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103210646 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.863542+0000", 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5812e-05 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965285+0000", 2024-04-18T21:51:53.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10174351600000001 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965365+0000", 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9589e-05 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=20, last_complete=30'7, committed=1, applied=1))", 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.765015+0000", 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3320173469999999, 2024-04-18T21:51:53.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20013165299999999, 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765015+0000", 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765015+0000", 2024-04-18T21:51:53.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765015+0000", 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0999999999999999e-07 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765018+0000", 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9340000000000002e-06 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765028+0000", 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4800000000000007e-06 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765032+0000", 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8210000000000001e-06 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.964972+0000", 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19993925300000001 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965146+0000", 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017465499999999999 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.331400146, 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299971873, 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.141 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1800000000000001e-07 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765635+0000", 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6450000000000001e-06 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765644+0000", 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.755e-06 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.865734+0000", 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10008987699999999 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065540+0000", 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19980606200000001 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065604+0000", 2024-04-18T21:51:53.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4115999999999994e-05 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.776822+0000", 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3202098849999997, 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.28932778799999997, 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776822+0000", 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776822+0000", 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776823+0000", 2024-04-18T21:51:53.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8699999999999995e-07 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776826+0000", 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4860000000000002e-06 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776836+0000", 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6549999999999999e-06 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.876933+0000", 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10009744 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.066102+0000", 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18916881299999999 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.066150+0000", 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7806999999999998e-05 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.2303373909999999, 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299440857, 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.13e-07 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866703+0000", 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866722+0000", 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9522999999999999e-05 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966809+0000", 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10008695400000001 2024-04-18T21:51:53.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166119+0000", 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19931021099999999 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166136+0000", 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6319000000000001e-05 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:78 3.15s0 3:ab33d1b5:::smithi09731216-26:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1160157509999999, 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1076682130000002, 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.153 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981019+0000", 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.447e-06 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981025+0000", 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6810000000000001e-06 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981044+0000", 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8856999999999999e-05 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981056+0000", 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1921e-05 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164029+0000", 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18297308600000001 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164033+0000", 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0369999999999996e-06 2024-04-18T21:51:53.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885490+0000", 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7214569900000001 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885771+0000", 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028064300000000002 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.088685+0000", 2024-04-18T21:51:53.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.202913551 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 31/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.065688+0000", 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.0313441670000003, 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20463684800000001, 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.158 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065688+0000", 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065688+0000", 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065690+0000", 2024-04-18T21:51:53.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8729999999999999e-06 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065701+0000", 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1677e-05 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065728+0000", 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.669e-05 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065752+0000", 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4196e-05 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269705+0000", 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20395275299999999 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.270325+0000", 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00061965900000000005 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.161 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.cs0 31/26 ECSubRead(tid=23, to_read={3:348e44d8:::smithi09731216-34:head=0,4096,0}, subchunks={3:348e44d8:::smithi09731216-34:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7243903100000004, 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214868642, 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372695+0000", 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3584000000000001e-05 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.472813+0000", 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10011788200000001 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.587035+0000", 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.114221857 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.587510+0000", 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047531899999999998 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.13s5 32/31 ECSubWrite(tid=14, reqid=client.4509.0:73, at_version=32'10, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:48.593977+0000", 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.5030555420000002, 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.204113025, 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593977+0000", 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593977+0000", 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593979+0000", 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1569999999999998e-06 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.594255+0000", 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027672299999999999 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.594300+0000", 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5086000000000002e-05 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.694435+0000", 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100134078 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.794654+0000", 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100219603 2024-04-18T21:51:53.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.794723+0000", 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8978000000000003e-05 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798007+0000", 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032838590000000001 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798090+0000", 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2540999999999995e-05 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s7 32/31 ECSubWrite(tid=11, reqid=client.4509.0:107, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:48.999453+0000", 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.0975790630000004, 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20156315299999999, 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.170 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.999453+0000", 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.999453+0000", 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.999453+0000", 2024-04-18T21:51:53.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3000000000000001e-07 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.999483+0000", 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0059000000000001e-05 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.999496+0000", 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2999999999999999e-05 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.999506+0000", 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6269999999999995e-06 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.199628+0000", 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20012148599999999 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.199642+0000", 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4722000000000001e-05 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.200955+0000", 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013124300000000001 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.201016+0000", 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1398999999999999e-05 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:133 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:53.174 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.0867497540000004, 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.393294378, 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 133 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:51:53.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010286+0000", 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012871+0000", 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025854699999999999 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012896+0000", 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5409e-05 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012911+0000", 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.484e-05 2024-04-18T21:51:53.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013086+0000", 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017422499999999999 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013437+0000", 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035166100000000002 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.016765+0000", 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033273949999999999 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.020122+0000", 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033576370000000001 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403436+0000", 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38331330800000002 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403577+0000", 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014114999999999999 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:134 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.0840482280000003, 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39119692299999997, 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.180 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 134 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012986+0000", 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0839999999999999e-06 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012995+0000", 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4159999999999993e-06 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013025+0000", 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0035999999999998e-05 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013046+0000", 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0720000000000002e-05 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.017993+0000", 2024-04-18T21:51:53.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049471020000000001 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.018258+0000", 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026496000000000001 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.019560+0000", 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001301373 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.020585+0000", 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010257910000000001 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403896+0000", 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38331068899999998 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.404181+0000", 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028475199999999999 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.185 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 33/26 ECSubWriteReply(tid=26, last_complete=32'10, committed=1, applied=1))", 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.120920+0000", 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.976111687, 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20035507799999999, 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.120920+0000", 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.120920+0000", 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.120922+0000", 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.187e-06 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.120931+0000", 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0799999999999995e-06 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.120951+0000", 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9848999999999999e-05 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.221048+0000", 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10009776300000001 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.321242+0000", 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10019343999999999 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.321275+0000", 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3759e-05 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.189 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:53.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:51:53.191 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:53.187+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:53.292 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:53.291+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:53.324 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:53.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:53.324 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:51:53.324 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:53.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:51:53.392 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:53.391+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:53.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:53.491+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:53.505 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:53.505 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:53.505 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:53.505 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:53.505 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:53.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:51:53.604 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:53.603+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.cs2 30/26 ECSubWrite(tid=16, reqid=client.4509.0:80, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.357836+0000", 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.3278085500000003, 2024-04-18T21:51:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101219042, 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357836+0000", 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357836+0000", 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357837+0000", 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.066e-06 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357853+0000", 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5255e-05 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357874+0000", 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.137e-05 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.357885+0000", 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0604e-05 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.458089+0000", 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100204521 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.458105+0000", 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5957e-05 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459001+0000", 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089603999999999997 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.459055+0000", 2024-04-18T21:51:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4228999999999999e-05 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.562752+0000", 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.1228932079999998, 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201174983, 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562752+0000", 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562752+0000", 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562752+0000", 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5199999999999997e-07 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562767+0000", 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4399999999999999e-05 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562778+0000", 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1225e-05 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.562784+0000", 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5060000000000001e-06 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763008+0000", 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200223922 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763022+0000", 2024-04-18T21:51:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3551e-05 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763889+0000", 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00086700299999999998 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763927+0000", 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7824000000000002e-05 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 30/26 ECSubWrite(tid=9, reqid=client.4509.0:95, at_version=0'0, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.966703+0000", 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.7189414960000002, 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103715603, 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966703+0000", 2024-04-18T21:51:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966703+0000", 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966704+0000", 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9399999999999993e-07 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966726+0000", 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1616000000000001e-05 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966745+0000", 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.863e-05 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966763+0000", 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7816e-05 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.066982+0000", 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10021948 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.066999+0000", 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7479999999999999e-05 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.070419+0000", 2024-04-18T21:51:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003419587 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.cs2 31/26 ECSubWrite(tid=22, reqid=client.4509.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=30'8))", 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.270184+0000", 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.4154607779999999, 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101561839, 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.270184+0000", 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.270184+0000", 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.270187+0000", 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6299999999999998e-06 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.270222+0000", 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5556000000000003e-05 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.270263+0000", 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0946999999999999e-05 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.270281+0000", 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8043e-05 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.370539+0000", 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100258109 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.370557+0000", 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.768e-05 2024-04-18T21:51:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.371669+0000", 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011122549999999999 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.371746+0000", 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6619000000000006e-05 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 31/26 ECSubWrite(tid=14, reqid=client.4509.0:115, at_version=31'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.274242+0000", 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.411403387, 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105789985, 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.274242+0000", 2024-04-18T21:51:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.274242+0000", 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.274242+0000", 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5700000000000001e-07 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.278008+0000", 2024-04-18T21:51:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037657979999999999 2024-04-18T21:51:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.278023+0000", 2024-04-18T21:51:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5114e-05 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.378106+0000", 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100083085 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.378229+0000", 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012252500000000001 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.378251+0000", 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1546e-05 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.379960+0000", 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017097569999999999 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.380032+0000", 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1303000000000005e-05 2024-04-18T21:51:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.11s6 32/31 ECSubWrite(tid=9, reqid=client.4509.0:127, at_version=32'5, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:48.899509+0000", 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.786136452, 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20351012600000001, 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899509+0000", 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899509+0000", 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899510+0000", 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.807e-06 2024-04-18T21:51:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899962+0000", 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045185800000000002 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899992+0000", 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9949000000000001e-05 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000095+0000", 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10010311299999999 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.100305+0000", 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10020960800000001 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.100322+0000", 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6657e-05 2024-04-18T21:51:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.102893+0000", 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025716609999999998 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.103019+0000", 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000125473 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.11s6 32/31 ECSubWrite(tid=10, reqid=client.4509.0:128, at_version=32'6, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.000150+0000", 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.6854944969999996, 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103113438, 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000150+0000", 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000150+0000", 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000152+0000", 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.093e-06 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000183+0000", 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1726000000000001e-05 2024-04-18T21:51:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000203+0000", 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9905999999999999e-05 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000217+0000", 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3687e-05 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.101515+0000", 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101298334 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.101543+0000", 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7826000000000001e-05 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.103199+0000", 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001655828 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.103264+0000", 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5037999999999996e-05 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 32/31 ECSubWrite(tid=11, reqid=client.4509.0:107, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.000376+0000", 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.6852690729999997, 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102907027, 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000376+0000", 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000376+0000", 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000377+0000", 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2699999999999999e-07 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000446+0000", 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9704999999999998e-05 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000467+0000", 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0822000000000002e-05 2024-04-18T21:51:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000488+0000", 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0409999999999999e-05 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000647+0000", 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000159512 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000661+0000", 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3867e-05 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.103181+0000", 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10252022600000001 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.103283+0000", 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000101758 2024-04-18T21:51:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.16s5 32/31 ECSubWrite(tid=12, reqid=client.4509.0:109, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.000517+0000", 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.6851275660000002, 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103393917, 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000517+0000", 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000517+0000", 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000518+0000", 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9599999999999998e-07 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000870+0000", 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035186499999999998 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000895+0000", 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5058999999999998e-05 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000915+0000", 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0205999999999999e-05 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.101176+0000", 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100260618 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.101205+0000", 2024-04-18T21:51:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9056000000000001e-05 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.103835+0000", 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026294719999999999 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.103911+0000", 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6744999999999998e-05 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.1s6 32/31 ECSubWrite(tid=13, reqid=client.4509.0:130, at_version=32'5, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.013132+0000", 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.6725127539999995, 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19062042000000001, 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013132+0000", 2024-04-18T21:51:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013132+0000", 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013134+0000", 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.486e-06 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013588+0000", 2024-04-18T21:51:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000453919 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013626+0000", 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8642e-05 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013651+0000", 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4598999999999999e-05 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.101424+0000", 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087773243000000001 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.101446+0000", 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1878999999999998e-05 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.203699+0000", 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102252647 2024-04-18T21:51:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.203753+0000", 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4005000000000002e-05 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.1s6 32/31 ECSubWrite(tid=14, reqid=client.4509.0:131, at_version=32'6, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.014216+0000", 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.6714293290000004, 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18987130699999999, 2024-04-18T21:51:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.014216+0000", 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.014216+0000", 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.014217+0000", 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4169999999999999e-06 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.014248+0000", 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1338999999999999e-05 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.014270+0000", 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1619000000000001e-05 2024-04-18T21:51:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.014290+0000", 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9656999999999998e-05 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.202919+0000", 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.188628935 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.202933+0000", 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4562999999999999e-05 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.204002+0000", 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010688150000000001 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.204087+0000", 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4962000000000004e-05 2024-04-18T21:51:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 32/31 ECSubWrite(tid=14, reqid=client.4509.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=32'8))", 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.098923+0000", 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.5867221210000002, 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20155577, 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.098923+0000", 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.098923+0000", 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.098923+0000", 2024-04-18T21:51:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9400000000000001e-07 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.098944+0000", 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0849e-05 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.098977+0000", 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3262000000000001e-05 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.099003+0000", 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5987e-05 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.299427+0000", 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200423396 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.299451+0000", 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3884999999999999e-05 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.300396+0000", 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094560599999999999 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.300479+0000", 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2391000000000004e-05 2024-04-18T21:51:53.744 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s3 33/31 ECSubWrite(tid=23, reqid=client.4509.0:59, at_version=33'10, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.212268+0000", 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.4733770359999996, 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10192248600000001, 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.212268+0000", 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.212268+0000", 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.212270+0000", 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1469999999999999e-06 2024-04-18T21:51:53.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.212314+0000", 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4085000000000002e-05 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.212354+0000", 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0151999999999998e-05 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.212379+0000", 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4493000000000001e-05 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.312672+0000", 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100293638 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.312697+0000", 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4533e-05 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.314087+0000", 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001390121 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.314190+0000", 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103317 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.750 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.16s5 33/31 ECSubWrite(tid=19, reqid=client.4509.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=32'3))", 2024-04-18T21:51:53.750 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.517359+0000", 2024-04-18T21:51:53.750 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.1682861390000001, 2024-04-18T21:51:53.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102208005, 2024-04-18T21:51:53.750 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.751 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.751 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.751 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.517359+0000", 2024-04-18T21:51:53.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.752 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.517359+0000", 2024-04-18T21:51:53.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.517360+0000", 2024-04-18T21:51:53.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3400000000000001e-06 2024-04-18T21:51:53.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.517424+0000", 2024-04-18T21:51:53.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4214999999999994e-05 2024-04-18T21:51:53.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.517453+0000", 2024-04-18T21:51:53.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8473e-05 2024-04-18T21:51:53.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.617565+0000", 2024-04-18T21:51:53.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100112175 2024-04-18T21:51:53.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.617823+0000", 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025840299999999999 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.617850+0000", 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6423000000000001e-05 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.619499+0000", 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016491310000000001 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.619567+0000", 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7844999999999998e-05 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.758 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s3 35/31 ECSubWrite(tid=51, reqid=client.4509.0:194, at_version=35'21, trim_to=0'0, roll_forward_to=34'19))", 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:52.582384+0000", 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.103260686, 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102318252, 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.582384+0000", 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.582384+0000", 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.582387+0000", 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5100000000000001e-06 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.582516+0000", 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012878100000000001 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.582558+0000", 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2268999999999998e-05 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.582586+0000", 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8481999999999999e-05 2024-04-18T21:51:53.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.682862+0000", 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100275488 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.682882+0000", 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0163e-05 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.684643+0000", 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017614359999999999 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.684703+0000", 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9123000000000003e-05 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.13s3 35/31 ECSubWrite(tid=52, reqid=client.4509.0:205, at_version=35'18, trim_to=0'0, roll_forward_to=33'16))", 2024-04-18T21:51:53.763 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:52.586113+0000", 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.099531861, 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10268488100000001, 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.586113+0000", 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.586113+0000", 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.586115+0000", 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2060000000000001e-06 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.586479+0000", 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036322500000000001 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.586521+0000", 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2942999999999999e-05 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.586549+0000", 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7268999999999999e-05 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.686723+0000", 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100174164 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.686737+0000", 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3929e-05 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.688703+0000", 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001966621 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.688798+0000", 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4524000000000005e-05 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s3 35/31 ECSubWrite(tid=53, reqid=client.4509.0:194, at_version=0'0, trim_to=0'0, roll_forward_to=35'21))", 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:52.688602+0000", 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.99704334500000003, 2024-04-18T21:51:53.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102008667, 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.688602+0000", 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.688602+0000", 2024-04-18T21:51:53.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.688604+0000", 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1600000000000001e-06 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.688632+0000", 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8666e-05 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.688684+0000", 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1715000000000002e-05 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.688703+0000", 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9253999999999999e-05 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.788942+0000", 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10023857799999999 2024-04-18T21:51:53.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.788958+0000", 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6436e-05 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.790569+0000", 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016103420000000001 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:52.790610+0000", 2024-04-18T21:51:53.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1516e-05 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 36/26 ECSubWrite(tid=36, reqid=client.4509.0:216, at_version=36'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.66709974100000002, 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30205913200000001, 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.35e-07 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018570+0000", 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.438e-05 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018583+0000", 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2928999999999999e-05 2024-04-18T21:51:53.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118702+0000", 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1001194 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218952+0000", 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10025022 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218966+0000", 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3709e-05 2024-04-18T21:51:53.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320525+0000", 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10155911099999999 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320604+0000", 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9147999999999995e-05 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 36/26 ECSubWrite(tid=37, reqid=client.4509.0:217, at_version=36'13, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.118770+0000", 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.566875138, 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20354913499999999, 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118770+0000", 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118770+0000", 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118771+0000", 2024-04-18T21:51:53.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.345e-06 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118799+0000", 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7997999999999999e-05 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118812+0000", 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3286e-05 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218952+0000", 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100139298 2024-04-18T21:51:53.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219853+0000", 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00090166699999999998 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219867+0000", 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3304e-05 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.322235+0000", 2024-04-18T21:51:53.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10236814900000001 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.322319+0000", 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4087999999999999e-05 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 36/26 ECSubWrite(tid=38, reqid=client.4509.0:218, at_version=36'14, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.219009+0000", 2024-04-18T21:51:53.782 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.466635472, 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103380338, 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219009+0000", 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219009+0000", 2024-04-18T21:51:53.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219011+0000", 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0359999999999999e-06 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219064+0000", 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3313000000000003e-05 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:53.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219087+0000", 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3302000000000002e-05 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219106+0000", 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9072999999999998e-05 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320335+0000", 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10122890399999999 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320347+0000", 2024-04-18T21:51:53.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1805e-05 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.322330+0000", 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019830389999999998 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.322390+0000", 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9865999999999999e-05 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.787 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:53.787 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:53.787 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:53.788 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:51:53.789 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:53.787+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:53.889 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:53.887+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:53.960 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:53.961 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:53.961 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:51:53.961 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:53.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:51:53.990 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:53.987+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:54.013 INFO:tasks.rados.rados.0.smithi097.stdout:90: done (7 left) 2024-04-18T21:51:54.013 INFO:tasks.rados.rados.0.smithi097.stdout:91: rmattr oid 33 current snap is 3 2024-04-18T21:51:54.013 INFO:tasks.rados.rados.0.smithi097.stdout:91: done (7 left) 2024-04-18T21:51:54.013 INFO:tasks.rados.rados.0.smithi097.stdout:92: delete oid 7 current snap is 3 2024-04-18T21:51:54.013 INFO:tasks.rados.rados.0.smithi097.stdout:84: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-18T21:51:54.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-18T21:51:54.038 INFO:tasks.rados.rados.0.smithi097.stdout:87: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-18T21:51:54.043 INFO:tasks.rados.rados.0.smithi097.stdout:89: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-18T21:51:54.076 INFO:tasks.rados.rados.0.smithi097.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-18T21:51:54.091 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:54.091+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:54.092 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:54.092 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:54.092 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:54.092 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:54.092 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:54.104 INFO:tasks.rados.rados.0.smithi097.stdout:88: finishing write tid 1 to smithi09731216-3 2024-04-18T21:51:54.104 INFO:tasks.rados.rados.0.smithi097.stdout:86: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-18T21:51:54.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:83: finishing rollback tid 0 to smithi09731216-9 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:83: finishing rollback tid 1 to smithi09731216-9 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:83: finishing rollback tid 2 to smithi09731216-9 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 9 v 14 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:88: finishing write tid 2 to smithi09731216-3 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:88: finishing write tid 3 to smithi09731216-3 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:88: finishing write tid 4 to smithi09731216-3 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:88: oid 3 updating version 0 to 23 2024-04-18T21:51:54.108 INFO:tasks.rados.rados.0.smithi097.stdout:88: oid 3 updating version 23 to 24 2024-04-18T21:51:54.109 INFO:tasks.rados.rados.0.smithi097.stdout:88: oid 3 updating version 24 to 25 2024-04-18T21:51:54.109 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 3 v 25 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-04-18T21:51:54.109 INFO:tasks.rados.rados.0.smithi097.stdout:88: left oid 3 (ObjNum 54 snap 3 seq_num 54) 2024-04-18T21:51:54.192 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:54.187+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.437153524999999, 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58281355499999998, 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.237 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 1 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798268+0000", 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5959999999999998e-06 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798283+0000", 2024-04-18T21:51:54.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5495999999999999e-05 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798322+0000", 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8899999999999997e-05 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798347+0000", 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5510000000000001e-05 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886734+0000", 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088386079000000006 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886749+0000", 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5019e-05 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.078589+0000", 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19184037900000001 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079052+0000", 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046354399999999998 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079551+0000", 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049870299999999997 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.083572+0000", 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040212609999999999 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.380910+0000", 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29733748900000001 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381009+0000", 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9025999999999997e-05 2024-04-18T21:51:54.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381079+0000", 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9553000000000003e-05 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.245 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.500344191, 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40660126499999999, 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 33 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735075+0000", 2024-04-18T21:51:54.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4300000000000001e-07 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735081+0000", 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8669999999999999e-06 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735094+0000", 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2813000000000001e-05 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735102+0000", 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4770000000000003e-06 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.737810+0000", 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027080089999999999 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.737953+0000", 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014291900000000001 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.738162+0000", 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020895500000000001 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.940717+0000", 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20255461299999999 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141518+0000", 2024-04-18T21:51:54.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20080082699999999 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141607+0000", 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9262e-05 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141676+0000", 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8579999999999997e-05 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.251 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.500303401, 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50698967299999997, 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 34 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735116+0000", 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735123+0000", 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3629999999999996e-06 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735156+0000", 2024-04-18T21:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3030000000000001e-05 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735167+0000", 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.13e-05 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939428+0000", 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20426096299999999 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939615+0000", 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018654199999999999 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939841+0000", 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022621800000000001 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141747+0000", 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201905318 2024-04-18T21:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.241958+0000", 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100211063 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242041+0000", 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3300000000000005e-05 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242105+0000", 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3895999999999999e-05 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.500228402000001, 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61088117099999995, 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.258 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 35 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735191+0000", 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3699999999999999e-07 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735266+0000", 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4856000000000003e-05 2024-04-18T21:51:54.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735279+0000", 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.36e-05 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735288+0000", 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0860000000000008e-06 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242298+0000", 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50700964999999998 2024-04-18T21:51:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242568+0000", 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027023100000000001 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242974+0000", 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040533399999999998 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246576+0000", 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036026919999999998 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345792+0000", 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099215503999999996 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345953+0000", 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161189 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346071+0000", 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118192 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.500115909000002, 2024-04-18T21:51:54.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61108955600000003, 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 36 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:54.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.23e-07 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735310+0000", 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7560000000000002e-06 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735323+0000", 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735337+0000", 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4049999999999999e-05 2024-04-18T21:51:54.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244243+0000", 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50890638099999996 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244422+0000", 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000178495 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244638+0000", 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216562 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.247950+0000", 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033121750000000001 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346232+0000", 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098281564000000002 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346324+0000", 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2042999999999999e-05 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346392+0000", 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8052999999999994e-05 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.270 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.500066889999999, 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71224671699999997, 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 37 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0999999999999998e-07 2024-04-18T21:51:54.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735358+0000", 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2639999999999999e-06 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735367+0000", 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735375+0000", 2024-04-18T21:51:54.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5360000000000004e-06 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245345+0000", 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50996983699999998 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245615+0000", 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027018099999999998 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246022+0000", 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000407394 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.248536+0000", 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025132330000000001 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447444+0000", 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198908419 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447532+0000", 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8010000000000006e-05 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447598+0000", 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6383000000000005e-05 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.276 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.870526940000001, 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.404608044, 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364892+0000", 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364893+0000", 2024-04-18T21:51:54.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1429999999999999e-06 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364969+0000", 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6136000000000004e-05 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.364981+0000", 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2099999999999999e-05 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465122+0000", 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10014120999999999 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465287+0000", 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000164499 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465554+0000", 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000267194 2024-04-18T21:51:54.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.465927+0000", 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000373171 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.467760+0000", 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018333749999999999 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769354+0000", 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.301593271 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769435+0000", 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1433000000000005e-05 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:35.769500+0000", 2024-04-18T21:51:54.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4511999999999995e-05 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.852036207000001, 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40557132600000001, 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.283 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 85 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383382+0000", 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383383+0000", 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1799999999999999e-06 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383389+0000", 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6030000000000004e-06 2024-04-18T21:51:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383400+0000", 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.059e-05 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383410+0000", 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.039e-05 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.485426+0000", 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10201608199999999 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.485677+0000", 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025043800000000002 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486126+0000", 2024-04-18T21:51:54.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00044914 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489043+0000", 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029174309999999998 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788775+0000", 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29973160999999998 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788881+0000", 2024-04-18T21:51:54.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106443 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.788954+0000", 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2418999999999999e-05 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.851996471, 2024-04-18T21:51:54.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40574237800000001, 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 86 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:54.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383422+0000", 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383423+0000", 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4700000000000001e-07 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383428+0000", 2024-04-18T21:51:54.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383437+0000", 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4630000000000007e-06 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383445+0000", 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.994e-06 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486731+0000", 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103285425 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.486864+0000", 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013336899999999999 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487072+0000", 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020769500000000001 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489073+0000", 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020010010000000001 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789030+0000", 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29995712499999999 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789104+0000", 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4257000000000004e-05 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789164+0000", 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0291999999999998e-05 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.851956917999999, 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40594649700000002, 2024-04-18T21:51:54.295 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 87 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383462+0000", 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5400000000000001e-07 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383535+0000", 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2879000000000005e-05 2024-04-18T21:51:54.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.383546+0000", 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1362999999999999e-05 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483681+0000", 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100134521 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487677+0000", 2024-04-18T21:51:54.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039962119999999999 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.487809+0000", 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013202900000000001 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.488004+0000", 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000195198 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.490235+0000", 2024-04-18T21:51:54.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002230073 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789259+0000", 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29902419899999999 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789354+0000", 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5443000000000001e-05 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.789408+0000", 2024-04-18T21:51:54.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3925999999999998e-05 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.751699791, 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40540477699999999, 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.301 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 88 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483719+0000", 2024-04-18T21:51:54.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483720+0000", 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5859999999999999e-06 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483731+0000", 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1090000000000001e-05 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483753+0000", 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1426000000000001e-05 2024-04-18T21:51:54.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.483765+0000", 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2013e-05 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489152+0000", 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053867120000000001 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489297+0000", 2024-04-18T21:51:54.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014540199999999999 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.489556+0000", 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025916099999999999 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.490845+0000", 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00128853 2024-04-18T21:51:54.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.888963+0000", 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39811801499999999 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.889050+0000", 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7583000000000005e-05 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:36.889123+0000", 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3258999999999998e-05 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.586666377, 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61732750999999997, 2024-04-18T21:51:54.307 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 117 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648753+0000", 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648758+0000", 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2024-04-18T21:51:54.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648768+0000", 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.414e-06 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648777+0000", 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.363e-06 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863479+0000", 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214701524 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863596+0000", 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117039 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863810+0000", 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021416800000000001 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.864846+0000", 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001036319 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265892+0000", 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40104641899999999 2024-04-18T21:51:54.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265995+0000", 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010209600000000001 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.266080+0000", 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5049000000000005e-05 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.313 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.929755137000001, 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.509317829, 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.314 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 158 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305665+0000", 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5230000000000001e-06 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305786+0000", 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012069 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305805+0000", 2024-04-18T21:51:54.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9836000000000001e-05 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305821+0000", 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6030000000000001e-05 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307593+0000", 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017715299999999999 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307838+0000", 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024551800000000002 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.308174+0000", 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033593400000000003 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413731+0000", 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105556865 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814825+0000", 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40109329500000002 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814911+0000", 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6138999999999996e-05 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814981+0000", 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0469000000000006e-05 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.319 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.929570081, 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50938204099999995, 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 159 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.320 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305849+0000", 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0559999999999999e-06 2024-04-18T21:51:54.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305856+0000", 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9600000000000003e-06 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305867+0000", 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0910000000000001e-05 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305879+0000", 2024-04-18T21:51:54.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.147e-05 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309071+0000", 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031920859999999998 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309303+0000", 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023236200000000001 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309671+0000", 2024-04-18T21:51:54.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036798000000000001 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416908+0000", 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107236578 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815085+0000", 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39817670700000002 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815165+0000", 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0196000000000003e-05 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815230+0000", 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5735999999999995e-05 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:54.325 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.929526778, 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50958771300000005, 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 160 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305898+0000", 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1359999999999996e-06 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305907+0000", 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5300000000000002e-06 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305915+0000", 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1869999999999997e-06 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310667+0000", 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047516800000000003 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310895+0000", 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000227875 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.311257+0000", 2024-04-18T21:51:54.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000361737 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416933+0000", 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10567686699999999 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815344+0000", 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39841079899999998 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815421+0000", 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7229000000000002e-05 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815479+0000", 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7973000000000002e-05 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:54.331 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.929492116, 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60955713600000005, 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 161 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:54.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305927+0000", 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306056+0000", 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012895300000000001 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306068+0000", 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306077+0000", 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9430000000000006e-06 2024-04-18T21:51:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412179+0000", 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106101709 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412267+0000", 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8251999999999995e-05 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412388+0000", 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121136 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420472+0000", 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080833510000000008 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915141+0000", 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49466944899999998 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915356+0000", 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215305 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915484+0000", 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012713500000000001 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.929328292999999, 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61031149600000001, 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.337 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 162 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306091+0000", 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306097+0000", 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6300000000000003e-06 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306106+0000", 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2199999999999998e-06 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306116+0000", 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8870000000000004e-06 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413869+0000", 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107753534 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414135+0000", 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026615200000000002 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414518+0000", 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038217299999999997 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420492+0000", 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059740879999999998 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915940+0000", 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49544827600000002 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916143+0000", 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000202752 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916402+0000", 2024-04-18T21:51:54.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025901100000000002 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.929291922999999, 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61064147300000005, 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.343 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 163 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.06e-07 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306136+0000", 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6999999999999997e-06 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306146+0000", 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8439999999999993e-06 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306154+0000", 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415593+0000", 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109439392 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415833+0000", 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000240101 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416223+0000", 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000389804 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420508+0000", 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042851699999999996 2024-04-18T21:51:54.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916537+0000", 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49602917499999999 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916669+0000", 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179899999999999 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916768+0000", 2024-04-18T21:51:54.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8879e-05 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.92925339, 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61096294100000004, 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.349 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 164 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306166+0000", 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306275+0000", 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109749 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306287+0000", 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1182999999999999e-05 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306298+0000", 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1487e-05 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417044+0000", 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110745818 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417282+0000", 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000237712 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417665+0000", 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000383194 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420523+0000", 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028578010000000001 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916880+0000", 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49635731900000002 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917054+0000", 2024-04-18T21:51:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017436200000000001 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917128+0000", 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3693000000000001e-05 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:54.355 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.929107394999999, 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70940268200000001, 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 165 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:54.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306312+0000", 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:54.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306317+0000", 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0699999999999997e-06 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306327+0000", 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7030000000000001e-06 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306349+0000", 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2160000000000001e-05 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419233+0000", 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112884159 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419499+0000", 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265865 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419851+0000", 2024-04-18T21:51:54.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035253999999999998 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.421040+0000", 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011886380000000001 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015628+0000", 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59458825299999996 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:54.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015677+0000", 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8470000000000002e-05 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015714+0000", 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7123999999999999e-05 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:54.361 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:54.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:51:54.364 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:54.359+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:54.422 INFO:tasks.rados.rados.0.smithi097.stdout:83: done (7 left) 2024-04-18T21:51:54.422 INFO:tasks.rados.rados.0.smithi097.stdout:84: done (6 left) 2024-04-18T21:51:54.422 INFO:tasks.rados.rados.0.smithi097.stdout:85: done (5 left) 2024-04-18T21:51:54.422 INFO:tasks.rados.rados.0.smithi097.stdout:86: done (4 left) 2024-04-18T21:51:54.422 INFO:tasks.rados.rados.0.smithi097.stdout:87: done (3 left) 2024-04-18T21:51:54.422 INFO:tasks.rados.rados.0.smithi097.stdout:88: done (2 left) 2024-04-18T21:51:54.422 INFO:tasks.rados.rados.0.smithi097.stdout:89: done (1 left) 2024-04-18T21:51:54.423 INFO:tasks.rados.rados.0.smithi097.stdout:92: done (0 left) 2024-04-18T21:51:54.423 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 29 current snap is 3 2024-04-18T21:51:54.423 INFO:tasks.rados.rados.0.smithi097.stdout:93: seq_num 56 ranges {475136=573440,1048576=376832} 2024-04-18T21:51:54.439 INFO:tasks.rados.rados.0.smithi097.stdout:93: writing smithi09731216-29 from 475136 to 1048576 tid 1 2024-04-18T21:51:54.446 INFO:tasks.rados.rados.0.smithi097.stdout:93: writing smithi09731216-29 from 1048576 to 1425408 tid 2 2024-04-18T21:51:54.446 INFO:tasks.rados.rados.0.smithi097.stdout:94: read oid 13 snap 0 2024-04-18T21:51:54.446 INFO:tasks.rados.rados.0.smithi097.stdout:94: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-18T21:51:54.446 INFO:tasks.rados.rados.0.smithi097.stdout:95: snap_remove snap 1 2024-04-18T21:51:54.464 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:54.459+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:54.497 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:54.497 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:54.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.497 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.2s5 37/31 ECSubRead(tid=43, to_read={3:44bf1f18:::smithi09731216-13:head=0,53248,0}, subchunks={3:44bf1f18:::smithi09731216-13:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:54.497 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:54.452880+0000", 2024-04-18T21:51:54.497 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.047456102999999999, 2024-04-18T21:51:54.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047506469000000003, 2024-04-18T21:51:54.497 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.452880+0000", 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.452880+0000", 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.452881+0000", 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7000000000000004e-07 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.452889+0000", 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6410000000000008e-06 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.452908+0000", 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8465999999999999e-05 2024-04-18T21:51:54.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.452921+0000", 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3548e-05 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.2s5 37/31 ECSubRead(tid=44, to_read={3:44bf1f18:::smithi09731216-13:head=0,53248,0}, subchunks={3:44bf1f18:::smithi09731216-13:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:54.453536+0000", 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.046799865000000003, 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046938196000000001, 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.500 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.453536+0000", 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.453536+0000", 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.453537+0000", 2024-04-18T21:51:54.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4900000000000005e-07 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.453553+0000", 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5719999999999999e-05 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.453570+0000", 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7224999999999999e-05 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:54.453582+0000", 2024-04-18T21:51:54.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1948e-05 2024-04-18T21:51:54.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.503 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.503 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:51:54.503 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 2 2024-04-18T21:51:54.503 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:54.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:51:54.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:51:54.565 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:54.563+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:54.631 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:54.631 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:51:54.632 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:51:54.632 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:51:54.632 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:54.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:51:54.666 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:54.663+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:54.714 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:51:54.714 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:51:54.767 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:54.767+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:54.770 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:51:54.770 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:51:54.792 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:51:54.792 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:51:54.810 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:51:54.810 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.2511230449999999, 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201960743, 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.818 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566449+0000", 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.77e-06 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566495+0000", 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5488999999999998e-05 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566536+0000", 2024-04-18T21:51:54.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0756000000000001e-05 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.666638+0000", 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102466 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766859+0000", 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10022078500000001 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766875+0000", 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5806000000000001e-05 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768312+0000", 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014373389999999999 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768408+0000", 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6331999999999999e-05 2024-04-18T21:51:54.822 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.4s4 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.1405963079999992, 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19094267400000001, 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676976+0000", 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9e-06 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677025+0000", 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8730000000000003e-05 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677075+0000", 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0405999999999998e-05 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677101+0000", 2024-04-18T21:51:54.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5777000000000001e-05 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.774116+0000", 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.097015016999999995 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.774145+0000", 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9009999999999998e-05 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867877+0000", 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093731420999999995 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867917+0000", 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0413000000000001e-05 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:118 3.17s0 3:eda737c1:::smithi09731216-40:head [append 0~671744,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:54.827 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.2128798160000001, 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1801127490000001, 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 118 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604693+0000", 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2970000000000002e-06 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.605995+0000", 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001302276 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606021+0000", 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5320000000000002e-05 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606036+0000", 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5133000000000001e-05 2024-04-18T21:51:54.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606232+0000", 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000196672 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606237+0000", 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7199999999999997e-06 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484281+0000", 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.87804420999999999 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484592+0000", 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031024299999999998 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.487837+0000", 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003245571 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490891+0000", 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030539719999999998 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784613+0000", 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.293722237 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784804+0000", 2024-04-18T21:51:54.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019009800000000001 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:119 3.17s0 3:eda737c1:::smithi09731216-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.2114979669999997, 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.179191986, 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:54.834 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 119 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:51:54.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606074+0000", 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.257e-06 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606080+0000", 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0100000000000001e-06 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606095+0000", 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5413e-05 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606104+0000", 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0899999999999994e-06 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606286+0000", 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000181552 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606287+0000", 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.026e-06 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493720+0000", 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88743291700000004 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493872+0000", 2024-04-18T21:51:54.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015190200000000001 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495103+0000", 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00123149 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.496855+0000", 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017518189999999999 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:51:54.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785096+0000", 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.28824129900000001 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785265+0000", 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016821100000000001 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:120 3.17s0 3:eda737c1:::smithi09731216-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:54.840 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.2114288369999997, 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4107226310000001, 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 120 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:51:54.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606143+0000", 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.16e-07 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606148+0000", 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4569999999999996e-06 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606163+0000", 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503e-05 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606175+0000", 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2036e-05 2024-04-18T21:51:54.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706293+0000", 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100117875 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706297+0000", 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.126e-06 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495621+0000", 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78932349599999996 2024-04-18T21:51:54.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495757+0000", 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013642600000000001 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785995+0000", 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29023745499999998 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.786127+0000", 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013212200000000001 2024-04-18T21:51:54.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.016864+0000", 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.23073769199999999 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=0, last_complete=32'1, committed=1, applied=1))", 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.2337532619999996, 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.112466288, 2024-04-18T21:51:54.846 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583819+0000", 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.604e-06 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583830+0000", 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0883000000000001e-05 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583857+0000", 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7053e-05 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683949+0000", 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100091869 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696264+0000", 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012315196 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696284+0000", 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9683000000000001e-05 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.7s0,GRANT,e:32,from:3(1),reservation_nonce:3)", 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.1683193100000002, 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14739324400000001, 2024-04-18T21:51:54.850 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:51:54.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649253+0000", 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0760000000000001e-06 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649265+0000", 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1523e-05 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649299+0000", 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3662999999999999e-05 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649316+0000", 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7197000000000002e-05 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796388+0000", 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14707189000000001 2024-04-18T21:51:54.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796474+0000", 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6038999999999994e-05 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796645+0000", 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017085599999999999 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.854 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 32/31 ECSubWrite(tid=12, reqid=client.4509.0:121, at_version=32'7, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.693738+0000", 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.1238323489999997, 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104018451, 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.693738+0000", 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.693738+0000", 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.693740+0000", 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.683e-06 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.694170+0000", 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00043018 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.694201+0000", 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0746e-05 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.794295+0000", 2024-04-18T21:51:54.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100094379 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.794449+0000", 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015417499999999999 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.794474+0000", 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4247e-05 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797719+0000", 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032449660000000002 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797757+0000", 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8074999999999997e-05 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.859 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.17s0 32/31 ECSubRead(tid=3, to_read={3:eda737c1:::smithi09731216-40:head=0,4096,0}, subchunks={3:eda737c1:::smithi09731216-40:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.0303878360000001, 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.210779366, 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:51:54.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787250+0000", 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7152999999999999e-05 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887353+0000", 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102647 2024-04-18T21:51:54.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997704+0000", 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110351217 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997962+0000", 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000258349 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.17s0 32/31 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.0210928319999999, 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200936119, 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796479+0000", 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8999999999999995e-07 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796491+0000", 2024-04-18T21:51:54.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2065000000000001e-05 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796502+0000", 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.116e-05 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796509+0000", 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7170000000000008e-06 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997338+0000", 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20082881799999999 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997414+0000", 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5568999999999994e-05 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.867 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.0209261119999997, 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200846881, 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796645+0000", 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8999999999999998e-07 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796647+0000", 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5239999999999999e-06 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796655+0000", 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796660+0000", 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4879999999999998e-06 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997463+0000", 2024-04-18T21:51:54.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20080266799999999 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997491+0000", 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8192e-05 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.0204055609999996, 2024-04-18T21:51:54.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20052022, 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:51:54.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797166+0000", 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5300000000000004e-07 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797170+0000", 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.233e-06 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797183+0000", 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3454e-05 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797191+0000", 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5769999999999996e-06 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997659+0000", 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20046852900000001 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997685+0000", 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5774000000000001e-05 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.7s0,GRANT,e:32,from:4(6),reservation_nonce:4)", 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.0200337470000003, 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20023016099999999, 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.875 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797544+0000", 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1670000000000002e-06 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797551+0000", 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9959999999999996e-06 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797574+0000", 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.321e-05 2024-04-18T21:51:54.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797587+0000", 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.235e-05 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997665+0000", 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200078382 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997695+0000", 2024-04-18T21:51:54.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0499999999999999e-05 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997767+0000", 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1556000000000004e-05 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:51:54.879 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.9305789469999999, 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110630505, 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886993+0000", 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.539e-06 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887021+0000", 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7529e-05 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887046+0000", 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4873000000000001e-05 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887060+0000", 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4324e-05 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997594+0000", 2024-04-18T21:51:54.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11053360800000001 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997622+0000", 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8632e-05 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.17s0 32/31 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.887927+0000", 2024-04-18T21:51:54.883 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.92964381, 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110115728, 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887927+0000", 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887927+0000", 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887928+0000", 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4399999999999999e-07 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887942+0000", 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4185999999999999e-05 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887957+0000", 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4951e-05 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887968+0000", 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1735e-05 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.998020+0000", 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110051488 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.998042+0000", 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2524e-05 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.889038+0000", 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.9285322750000002, 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.108512738, 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889038+0000", 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889038+0000", 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889039+0000", 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.63e-07 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889042+0000", 2024-04-18T21:51:54.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.241e-06 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889051+0000", 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0389999999999993e-06 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889059+0000", 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5190000000000003e-06 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997538+0000", 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10847904 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997551+0000", 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3436e-05 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.11s7 32/31 ECSubWrite(tid=9, reqid=client.4509.0:127, at_version=32'5, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:54.891 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.902577+0000", 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.9149932449999998, 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104903685, 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902577+0000", 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902577+0000", 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902580+0000", 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3769999999999999e-06 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902811+0000", 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023130399999999999 2024-04-18T21:51:54.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902856+0000", 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4993000000000001e-05 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.002953+0000", 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10009733699999999 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.003129+0000", 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017522999999999999 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.003159+0000", 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0382000000000002e-05 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.007383+0000", 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042242959999999998 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.007481+0000", 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7765999999999999e-05 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.10s5 33/31 ECSubWrite(tid=16, reqid=client.4509.0:139, at_version=33'7, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.310269+0000", 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.5073015749999996, 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103025701, 2024-04-18T21:51:54.896 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310269+0000", 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310269+0000", 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310270+0000", 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7400000000000003e-07 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310564+0000", 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029396000000000001 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310580+0000", 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6917000000000001e-05 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310596+0000", 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6019e-05 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310706+0000", 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109234 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310723+0000", 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6833999999999998e-05 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.413255+0000", 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102532847 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.413295+0000", 2024-04-18T21:51:54.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9316000000000001e-05 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s5 33/31 ECSubWrite(tid=23, reqid=client.4509.0:146, at_version=33'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.524136+0000", 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.2934346430000003, 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103402618, 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.524136+0000", 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.524136+0000", 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.524137+0000", 2024-04-18T21:51:54.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3829999999999999e-06 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.524232+0000", 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4881000000000002e-05 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.524248+0000", 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6220999999999999e-05 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.524259+0000", 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0756e-05 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.624677+0000", 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100417493 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.624707+0000", 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0531000000000003e-05 2024-04-18T21:51:54.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.627456+0000", 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027483439999999998 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.627539+0000", 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3009000000000004e-05 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s5 33/31 ECSubWrite(tid=24, reqid=client.4509.0:147, at_version=33'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.526901+0000", 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.2906699420000001, 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10323306, 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.526901+0000", 2024-04-18T21:51:54.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.526901+0000", 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.526902+0000", 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.793e-06 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.526981+0000", 2024-04-18T21:51:54.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8373000000000003e-05 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.527085+0000", 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104112 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.527137+0000", 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1978000000000003e-05 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.625735+0000", 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.098598401000000002 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.625764+0000", 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8734999999999998e-05 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.630073+0000", 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043087450000000001 2024-04-18T21:51:54.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.630134+0000", 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0922999999999999e-05 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:51:54.910 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:51:54.911 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:51:54.912 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:51:54.912 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:51:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:51:54.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:51:54.913 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:54.911+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:54.914 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:51:54.914 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:51:54.914 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:51:54.915 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:51:54.927 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:51:54.928 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:51:54.956 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:51:54.957 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:51:54.980 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:51:54.980 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:51:55.003 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:51:55.003 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:51:55.012 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:55.007+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:55.022 INFO:tasks.rados.rados.0.smithi097.stdout:95: done (2 left) 2024-04-18T21:51:55.022 INFO:tasks.rados.rados.0.smithi097.stdout:96: copy_from oid 34 from oid 37 current snap is 3 2024-04-18T21:51:55.022 INFO:tasks.rados.rados.0.smithi097.stdout:97: read oid 11 snap 0 2024-04-18T21:51:55.022 INFO:tasks.rados.rados.0.smithi097.stdout:97: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-18T21:51:55.022 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 38 current snap is 3 2024-04-18T21:51:55.022 INFO:tasks.rados.rados.0.smithi097.stdout:98: seq_num 57 ranges {622592=589824,1212416=376832} 2024-04-18T21:51:55.023 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is set 2024-04-18T21:51:55.033 INFO:tasks.rados.rados.0.smithi097.stdout:98: writing smithi09731216-38 from 622592 to 1212416 tid 1 2024-04-18T21:51:55.036 INFO:tasks.rados.rados.0.smithi097.stdout:98: writing smithi09731216-38 from 1212416 to 1589248 tid 2 2024-04-18T21:51:55.037 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 47 current snap is 3 2024-04-18T21:51:55.037 INFO:tasks.rados.rados.0.smithi097.stdout:94: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-18T21:51:55.045 INFO:tasks.rados.rados.0.smithi097.stdout:93: finishing write tid 1 to smithi09731216-29 2024-04-18T21:51:55.045 INFO:tasks.rados.rados.0.smithi097.stdout:93: finishing write tid 2 to smithi09731216-29 2024-04-18T21:51:55.045 INFO:tasks.rados.rados.0.smithi097.stdout:93: finishing write tid 3 to smithi09731216-29 2024-04-18T21:51:55.045 INFO:tasks.rados.rados.0.smithi097.stdout:99: seq_num 58 ranges {786432=606208,1392640=376832} 2024-04-18T21:51:55.052 INFO:tasks.rados.rados.0.smithi097.stdout:99: writing smithi09731216-47 from 786432 to 1392640 tid 1 2024-04-18T21:51:55.055 INFO:tasks.rados.rados.0.smithi097.stdout:99: writing smithi09731216-47 from 1392640 to 1769472 tid 2 2024-04-18T21:51:55.055 INFO:tasks.rados.rados.0.smithi097.stdout:94: done (5 left) 2024-04-18T21:51:55.055 INFO:tasks.rados.rados.0.smithi097.stdout:100: copy_from oid 42 from oid 4 current snap is 3 2024-04-18T21:51:55.055 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 20 current snap is 3 2024-04-18T21:51:55.055 INFO:tasks.rados.rados.0.smithi097.stdout:101: seq_num 59 ranges {327680=622592,950272=376832} 2024-04-18T21:51:55.060 INFO:tasks.rados.rados.0.smithi097.stdout:101: writing smithi09731216-20 from 327680 to 950272 tid 1 2024-04-18T21:51:55.062 INFO:tasks.rados.rados.0.smithi097.stdout:101: writing smithi09731216-20 from 950272 to 1327104 tid 2 2024-04-18T21:51:55.063 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 19 current snap is 3 2024-04-18T21:51:55.063 INFO:tasks.rados.rados.0.smithi097.stdout:102: seq_num 60 ranges {311296=638976,950272=376832} 2024-04-18T21:51:55.067 INFO:tasks.rados.rados.0.smithi097.stdout:102: writing smithi09731216-19 from 311296 to 950272 tid 1 2024-04-18T21:51:55.070 INFO:tasks.rados.rados.0.smithi097.stdout:102: writing smithi09731216-19 from 950272 to 1327104 tid 2 2024-04-18T21:51:55.070 INFO:tasks.rados.rados.0.smithi097.stdout:103: rollback oid 17 current snap is 3 2024-04-18T21:51:55.070 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 17 to 0 2024-04-18T21:51:55.070 INFO:tasks.rados.rados.0.smithi097.stdout:104: delete oid 45 current snap is 3 2024-04-18T21:51:55.086 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:55.086 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:55.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:51:55.101 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:51:55.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-18T21:51:55.113 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:55.111+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:55.214 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:55.211+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:55.227 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:55.227 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:55.232 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:55.231+0000 7f3d90efa640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:55.232 INFO:tasks.rados.rados.0.smithi097.stdout:93: finishing write tid 4 to smithi09731216-29 2024-04-18T21:51:55.232 INFO:tasks.rados.rados.0.smithi097.stdout:93: oid 29 updating version 0 to 29 2024-04-18T21:51:55.233 INFO:tasks.rados.rados.0.smithi097.stdout:93: oid 29 updating version 29 to 30 2024-04-18T21:51:55.233 INFO:tasks.rados.rados.0.smithi097.stdout:93: oid 29 updating version 30 to 31 2024-04-18T21:51:55.233 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 29 v 31 (ObjNum 56 snap 3 seq_num 56) dirty exists 2024-04-18T21:51:55.233 INFO:tasks.rados.rados.0.smithi097.stdout:93: left oid 29 (ObjNum 56 snap 3 seq_num 56) 2024-04-18T21:51:55.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:51:55.315 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:55.315+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:55.337 INFO:tasks.rados.rados.0.smithi097.stdout:96: finishing copy_from racing read to smithi09731216-34 2024-04-18T21:51:55.338 INFO:tasks.rados.rados.0.smithi097.stdout:93: done (9 left) 2024-04-18T21:51:55.338 INFO:tasks.rados.rados.0.smithi097.stdout:104: done (8 left) 2024-04-18T21:51:55.338 INFO:tasks.rados.rados.0.smithi097.stdout:105: read oid 7 snap -1 2024-04-18T21:51:55.338 INFO:tasks.rados.rados.0.smithi097.stdout:105: expect deleted 2024-04-18T21:51:55.338 INFO:tasks.rados.rados.0.smithi097.stdout:106: rollback oid 35 current snap is 3 2024-04-18T21:51:55.338 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 35 to 0 2024-04-18T21:51:55.338 INFO:tasks.rados.rados.0.smithi097.stdout:107: delete oid 6 current snap is 3 2024-04-18T21:51:55.364 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:55.365 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:55.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:51:55.416 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:55.415+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:55.427 INFO:tasks.rados.rados.0.smithi097.stdout:97: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-18T21:51:55.469 INFO:tasks.rados.rados.0.smithi097.stdout:100: finishing copy_from racing read to smithi09731216-42 2024-04-18T21:51:55.517 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:55.515+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:55.539 INFO:tasks.rados.rados.0.smithi097.stdout:96: finishing copy_from to smithi09731216-34 2024-04-18T21:51:55.539 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 34 v 4 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-18T21:51:55.549 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:55.549 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:55.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:51:55.617 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:55.615+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:55.682 INFO:tasks.rados.rados.0.smithi097.stdout:98: finishing write tid 1 to smithi09731216-38 2024-04-18T21:51:55.682 INFO:tasks.rados.rados.0.smithi097.stdout:106: finishing rollback tid 0 to smithi09731216-35 2024-04-18T21:51:55.683 INFO:tasks.rados.rados.0.smithi097.stdout:98: finishing write tid 2 to smithi09731216-38 2024-04-18T21:51:55.718 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:51:55.719+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:51:55.734 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:55.734 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:55.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:51:55.772 INFO:tasks.rados.rados.0.smithi097.stdout:100: finishing copy_from to smithi09731216-42 2024-04-18T21:51:55.772 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 42 v 13 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-18T21:51:55.774 INFO:tasks.rados.rados.0.smithi097.stdout:106: finishing rollback tid 1 to smithi09731216-35 2024-04-18T21:51:55.782 INFO:tasks.rados.rados.0.smithi097.stdout:106: finishing rollback tid 2 to smithi09731216-35 2024-04-18T21:51:55.782 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 35 v 13 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-18T21:51:55.783 INFO:tasks.rados.rados.0.smithi097.stdout:98: finishing write tid 3 to smithi09731216-38 2024-04-18T21:51:55.784 INFO:tasks.rados.rados.0.smithi097.stdout:101: finishing write tid 1 to smithi09731216-20 2024-04-18T21:51:55.785 INFO:tasks.rados.rados.0.smithi097.stdout:101: finishing write tid 2 to smithi09731216-20 2024-04-18T21:51:55.819 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:55.815+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:55.887 INFO:tasks.rados.rados.0.smithi097.stdout:98: finishing write tid 4 to smithi09731216-38 2024-04-18T21:51:55.887 INFO:tasks.rados.rados.0.smithi097.stdout:98: oid 38 updating version 0 to 33 2024-04-18T21:51:55.887 INFO:tasks.rados.rados.0.smithi097.stdout:98: oid 38 updating version 33 to 34 2024-04-18T21:51:55.888 INFO:tasks.rados.rados.0.smithi097.stdout:98: oid 38 updating version 34 to 35 2024-04-18T21:51:55.888 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 38 v 35 (ObjNum 57 snap 3 seq_num 57) dirty exists 2024-04-18T21:51:55.888 INFO:tasks.rados.rados.0.smithi097.stdout:98: left oid 38 (ObjNum 57 snap 3 seq_num 57) 2024-04-18T21:51:55.915 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:55.915 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:55.920 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:55.915+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:55.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:51:56.020 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:56.019+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:56.034 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.0 to 1 (10000) 2024-04-18T21:51:56.058 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:56.058 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:56.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-18T21:51:56.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:51:56.153 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:56.151+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:56.222 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:56.219+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:56.239 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:56.239 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:56.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:51:56.292 INFO:tasks.rados.rados.0.smithi097.stdout:101: finishing write tid 3 to smithi09731216-20 2024-04-18T21:51:56.294 INFO:tasks.rados.rados.0.smithi097.stdout:103: finishing rollback tid 0 to smithi09731216-17 2024-04-18T21:51:56.323 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:56.319+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:56.373 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:56.373 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:56.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:51:56.399 INFO:tasks.rados.rados.0.smithi097.stdout:103: finishing rollback tid 1 to smithi09731216-17 2024-04-18T21:51:56.424 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:56.423+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:56.500 INFO:tasks.rados.rados.0.smithi097.stdout:103: finishing rollback tid 2 to smithi09731216-17 2024-04-18T21:51:56.500 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 17 v 43 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-18T21:51:56.502 INFO:tasks.rados.rados.0.smithi097.stdout:101: finishing write tid 4 to smithi09731216-20 2024-04-18T21:51:56.502 INFO:tasks.rados.rados.0.smithi097.stdout:101: oid 20 updating version 0 to 39 2024-04-18T21:51:56.502 INFO:tasks.rados.rados.0.smithi097.stdout:101: oid 20 version 39 is already newer than 38 2024-04-18T21:51:56.502 INFO:tasks.rados.rados.0.smithi097.stdout:101: oid 20 version 39 is already newer than 37 2024-04-18T21:51:56.503 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 20 v 39 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-04-18T21:51:56.503 INFO:tasks.rados.rados.0.smithi097.stdout:101: left oid 20 (ObjNum 59 snap 3 seq_num 59) 2024-04-18T21:51:56.513 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:56.513 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:56.525 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:56.523+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:56.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:51:56.549 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:56.543+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:56.603 INFO:tasks.rados.rados.0.smithi097.stdout:96: done (10 left) 2024-04-18T21:51:56.603 INFO:tasks.rados.rados.0.smithi097.stdout:97: done (9 left) 2024-04-18T21:51:56.603 INFO:tasks.rados.rados.0.smithi097.stdout:98: done (8 left) 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:100: done (7 left) 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:101: done (6 left) 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:103: done (5 left) 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:105: done (4 left) 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:106: done (3 left) 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:107: done (2 left) 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:108: read oid 17 snap 0 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:108: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:109: rollback oid 40 current snap is 3 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 40 to 0 2024-04-18T21:51:56.604 INFO:tasks.rados.rados.0.smithi097.stdout:110: delete oid 8 current snap is 3 2024-04-18T21:51:56.626 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:56.623+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:56.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:51:56.727 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:56.727+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:56.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:56.823+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:56.850 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:56.850 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:56.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:51:56.915 INFO:tasks.rados.rados.0.smithi097.stdout:109: finishing rollback tid 0 to smithi09731216-40 2024-04-18T21:51:56.916 INFO:tasks.rados.rados.0.smithi097.stdout:109: finishing rollback tid 1 to smithi09731216-40 2024-04-18T21:51:56.927 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:56.923+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:56.986 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:56.986 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:57.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:51:57.011 INFO:tasks.rados.rados.0.smithi097.stdout:110: done (4 left) 2024-04-18T21:51:57.011 INFO:tasks.rados.rados.0.smithi097.stdout:111: setattr oid 3 current snap is 3 2024-04-18T21:51:57.014 INFO:tasks.rados.rados.0.smithi097.stdout:112: snap_create 2024-04-18T21:51:57.016 INFO:tasks.rados.rados.0.smithi097.stdout:109: finishing rollback tid 2 to smithi09731216-40 2024-04-18T21:51:57.016 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-18T21:51:57.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:51:57.028 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:57.023+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:57.040 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.1 to 1 (10000) 2024-04-18T21:51:57.044 INFO:tasks.rados.rados.0.smithi097.stdout:109: done (5 left) 2024-04-18T21:51:57.044 INFO:tasks.rados.rados.0.smithi097.stdout:112: done (4 left) 2024-04-18T21:51:57.045 INFO:tasks.rados.rados.0.smithi097.stdout:113: copy_from oid 39 from oid 40 current snap is 4 2024-04-18T21:51:57.045 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 18 current snap is 4 2024-04-18T21:51:57.045 INFO:tasks.rados.rados.0.smithi097.stdout:114: seq_num 64 ranges {245760=704512,950272=376832} 2024-04-18T21:51:57.053 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.053 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.053 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.053 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.054 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.055 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.056 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.057 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.058 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.043+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.059 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.060 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.061 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.062 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.062 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.062 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.062 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.062 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.062 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.062 INFO:tasks.rados.rados.0.smithi097.stdout:114: writing smithi09731216-18 from 245760 to 950272 tid 1 2024-04-18T21:51:57.063 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.063 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.063 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.063 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.047+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.063 INFO:tasks.rados.rados.0.smithi097.stdout:114: writing smithi09731216-18 from 950272 to 1327104 tid 2 2024-04-18T21:51:57.063 INFO:tasks.rados.rados.0.smithi097.stdout:115: snap_remove snap 3 2024-04-18T21:51:57.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:51:57.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-18T21:51:57.129 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:57.127+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:57.143 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:51:57.144 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.150 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.151 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.152 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.153 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.154 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.155 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.156 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.157 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.158 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.159 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.159 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.159 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.159 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.159 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.143+0000 7f87389b3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.165 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.166 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.167 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.168 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.169 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.170 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.147+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.171 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.172 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.173 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.174 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.151+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:51:57.177 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:51:57.178 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:51:57.212 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:51:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:51:57.229 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:57.227+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:57.245 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:51:57.245 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:51:57.266 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:51:57.267 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:51:57.303 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:51:57.303 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:51:57.314 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 29/26 ECSubWriteReply(tid=1, last_complete=29'2, committed=1, applied=1))", 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.034125+0000", 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.278699823, 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.108726638, 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034125+0000", 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034125+0000", 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034125+0000", 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0699999999999998e-07 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034128+0000", 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1920000000000001e-06 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034136+0000", 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5950000000000001e-06 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034141+0000", 2024-04-18T21:51:57.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4670000000000003e-06 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.142460+0000", 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.108318753 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.142851+0000", 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00039122400000000002 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.318 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 29/26 ECSubWriteReply(tid=2, last_complete=29'3, committed=1, applied=1))", 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.278671985999999, 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20866285400000001, 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.319 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034153+0000", 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3099999999999999e-07 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034154+0000", 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.838e-06 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034160+0000", 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3979999999999998e-06 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034179+0000", 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8751000000000001e-05 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242502+0000", 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.208323065 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242815+0000", 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031357100000000001 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 29/26 ECSubWriteReply(tid=3, last_complete=29'4, committed=1, applied=1))", 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.278637777, 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20991810899999999, 2024-04-18T21:51:57.323 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5400000000000002e-07 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034189+0000", 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.956e-06 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034196+0000", 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0509999999999999e-06 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034210+0000", 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4086e-05 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242874+0000", 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.208664241 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.244105+0000", 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001230521 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.bs4 29/26 ECSubWrite(tid=8, reqid=client.4509.0:70, at_version=29'3, trim_to=0'0, roll_forward_to=29'2))", 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.047551+0000", 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.265273755000001, 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104585065, 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.327 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.047551+0000", 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.047551+0000", 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.047552+0000", 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1429999999999999e-06 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.047920+0000", 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036791200000000002 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.047937+0000", 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7544e-05 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.047949+0000", 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2052000000000001e-05 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.048065+0000", 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011620600000000001 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.048078+0000", 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2201000000000001e-05 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.152065+0000", 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.103987334 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.152136+0000", 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0673000000000006e-05 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.331 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.bs4 29/26 ECSubWrite(tid=9, reqid=client.4509.0:71, at_version=29'4, trim_to=0'0, roll_forward_to=29'2))", 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.051612+0000", 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.261212549, 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.103854952, 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051612+0000", 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051612+0000", 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051612+0000", 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2199999999999999e-07 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051627+0000", 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4986e-05 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051637+0000", 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6579999999999997e-06 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051643+0000", 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4629999999999998e-06 2024-04-18T21:51:57.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.151788+0000", 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100144783 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.151801+0000", 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2696e-05 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.155409+0000", 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036081479999999998 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.155467+0000", 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7796e-05 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.fs1 30/26 ECSubWrite(tid=141, reqid=client.4509.0:88, at_version=30'3, trim_to=0'0, roll_forward_to=28'2))", 2024-04-18T21:51:57.336 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.061043+0000", 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.251781548, 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20224404100000001, 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061043+0000", 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.337 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061043+0000", 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061043+0000", 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5699999999999999e-07 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061314+0000", 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027134499999999998 2024-04-18T21:51:57.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061326+0000", 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1814000000000001e-05 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.161407+0000", 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10008054800000001 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.261648+0000", 2024-04-18T21:51:57.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100241765 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.261663+0000", 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4545999999999999e-05 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.263249+0000", 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001585524 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.263287+0000", 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8342e-05 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.fs1 30/26 ECSubWrite(tid=142, reqid=client.4509.0:89, at_version=30'4, trim_to=0'0, roll_forward_to=28'2))", 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.161441+0000", 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.151383284, 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10421773500000001, 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.341 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.161441+0000", 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.161441+0000", 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.161442+0000", 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9400000000000005e-07 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.161463+0000", 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1248000000000001e-05 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.161475+0000", 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1739999999999999e-05 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.161484+0000", 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8009999999999993e-06 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.262520+0000", 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101035926 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.262527+0000", 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8399999999999995e-06 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:57.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.265622+0000", 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00309472 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.265659+0000", 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6766e-05 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.0s0 30/26 ECSubReadReply(tid=6, attrs_read=0))", 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.345541+0000", 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.967283460999999, 2024-04-18T21:51:57.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200690538, 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345541+0000", 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345541+0000", 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345542+0000", 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3960000000000001e-06 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345567+0000", 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.472e-05 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345590+0000", 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2690000000000001e-05 2024-04-18T21:51:57.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.445667+0000", 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007769900000001 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.545898+0000", 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100231019 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.546231+0000", 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033301400000000002 2024-04-18T21:51:57.348 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:94 3.0s0 3:033c99c3:::smithi09731216-32:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.756974549000001, 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40893576500000001, 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555853+0000", 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1889999999999999e-06 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556617+0000", 2024-04-18T21:51:57.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076362200000000004 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556649+0000", 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2478999999999999e-05 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556665+0000", 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5837000000000001e-05 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556826+0000", 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016137200000000001 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.557241+0000", 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00041440699999999999 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.559642+0000", 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024011430000000001 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563415+0000", 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003773233 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964577+0000", 2024-04-18T21:51:57.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40116224099999997 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964786+0000", 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000208242 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:95 3.0s0 3:033c99c3:::smithi09731216-32:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:57.355 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.756121452, 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858905099999998, 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 95 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556705+0000", 2024-04-18T21:51:57.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6360000000000001e-06 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556715+0000", 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.066e-05 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556734+0000", 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8323999999999999e-05 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556745+0000", 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1240000000000001e-05 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561327+0000", 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0045823330000000001 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561583+0000", 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000255732 2024-04-18T21:51:57.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.562607+0000", 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010240990000000001 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563867+0000", 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012605439999999999 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965078+0000", 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40121052699999998 2024-04-18T21:51:57.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965292+0000", 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021395600000000001 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:96 3.0s0 3:033c99c3:::smithi09731216-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.75602391, 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42323290800000002, 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.361 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 96 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556802+0000", 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7400000000000001e-06 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556808+0000", 2024-04-18T21:51:57.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.993e-06 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556828+0000", 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9364e-05 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.656965+0000", 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10013702300000001 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762039+0000", 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105074903 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762350+0000", 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031089399999999998 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967780+0000", 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20542997700000001 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.972542+0000", 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004761932 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.979990+0000", 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0074475289999999996 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.980033+0000", 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3553000000000001e-05 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.366 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.4s2 30/26 ECSubWrite(tid=10, reqid=client.4509.0:97, at_version=30'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.860362+0000", 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.452462576, 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20550668799999999, 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860362+0000", 2024-04-18T21:51:57.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860362+0000", 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860363+0000", 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9399999999999993e-07 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860513+0000", 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014979199999999999 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860538+0000", 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5286000000000002e-05 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860552+0000", 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.385e-05 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.960935+0000", 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100383132 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.960960+0000", 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5179000000000001e-05 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065868+0000", 2024-04-18T21:51:57.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104908455 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.4s2 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.860641+0000", 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.452183128, 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.205319586, 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860641+0000", 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860641+0000", 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860642+0000", 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860660+0000", 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.859e-05 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860675+0000", 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5119999999999999e-05 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860685+0000", 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9729999999999992e-06 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.062389+0000", 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201703938 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.062420+0000", 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0877000000000003e-05 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065961+0000", 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035406280000000001 2024-04-18T21:51:57.374 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 30/26 ECSubWriteReply(tid=7, last_complete=30'5, committed=1, applied=1))", 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.861182+0000", 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.451642742000001, 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10363947699999999, 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.375 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.861182+0000", 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.861182+0000", 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.861182+0000", 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6000000000000003e-07 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.861187+0000", 2024-04-18T21:51:57.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6369999999999998e-06 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.861199+0000", 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1473e-05 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.961675+0000", 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100476064 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964352+0000", 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002677727 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964821+0000", 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046881599999999999 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.0s0 32/31 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:48.589477+0000", 2024-04-18T21:51:57.378 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 8.7233472439999993, 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10397579, 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.589477+0000", 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.589477+0000", 2024-04-18T21:51:57.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.589478+0000", 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0100000000000004e-07 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.589489+0000", 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1233000000000001e-05 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.589502+0000", 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2454e-05 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.689587+0000", 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100085417 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.693316+0000", 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037290660000000001 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.693453+0000", 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013681899999999999 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDScrubReserve(3.0s0,GRANT,e:35,from:9(6),reservation_nonce:9)", 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.377622+0000", 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.9352019760000001, 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200381114, 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.377622+0000", 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.377622+0000", 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.377624+0000", 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.207e-06 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.377629+0000", 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2530000000000004e-06 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.377650+0000", 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0743000000000001e-05 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.377666+0000", 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6232999999999999e-05 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.577793+0000", 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200126946 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.577866+0000", 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3252999999999998e-05 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.578004+0000", 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000137479 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:222 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.2999423209999996, 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30521668800000001, 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.386 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 222 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.387 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012885+0000", 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7760000000000002e-06 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012901+0000", 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6563e-05 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012934+0000", 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2719999999999998e-05 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012952+0000", 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7730000000000001e-05 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013179+0000", 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022670500000000001 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013200+0000", 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1466000000000001e-05 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114024+0000", 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100823616 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114179+0000", 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015544500000000001 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.318099+0000", 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203919667 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:223 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.2998218919999998, 2024-04-18T21:51:57.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40441434900000001, 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 223 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013004+0000", 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5400000000000001e-06 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013015+0000", 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0519999999999999e-05 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013077+0000", 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2848999999999994e-05 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013098+0000", 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1083000000000001e-05 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013359+0000", 2024-04-18T21:51:57.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000260408 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013383+0000", 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3867e-05 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114847+0000", 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10146374399999999 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114996+0000", 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149865 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.417417+0000", 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30242047300000002 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.396 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:224 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.2996917999999997, 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40564824999999999, 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 224 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013134+0000", 2024-04-18T21:51:57.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.387e-06 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013144+0000", 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0273e-05 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013166+0000", 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1857000000000001e-05 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013182+0000", 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5645999999999999e-05 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113542+0000", 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10035996599999999 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113552+0000", 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0492999999999999e-05 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215576+0000", 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102023869 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215730+0000", 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000153635 2024-04-18T21:51:57.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.418781+0000", 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203051124 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.0s0 36/31 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.217600+0000", 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.095224097, 2024-04-18T21:51:57.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.199856491, 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.217600+0000", 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.217600+0000", 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.217600+0000", 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.217600+0000", 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.316268+0000", 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098668031000000003 2024-04-18T21:51:57.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.416410+0000", 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100142077 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.416588+0000", 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017764199999999999 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.417457+0000", 2024-04-18T21:51:57.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00086874100000000002 2024-04-18T21:51:57.406 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.406 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.406 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.406 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.406 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.406 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:57.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:51:57.409 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:57.403+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:57.449 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:51:57.449 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:51:57.476 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:51:57.476 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:51:57.505 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:51:57.505 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:51:57.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:57.507+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:57.556 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.bs3 40/26 ECSubWrite(tid=50, reqid=client.4509.0:291, at_version=0'0, trim_to=0'0, roll_forward_to=40'20))", 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.556128+0000", 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.00095950699999999996, 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001021859, 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.556128+0000", 2024-04-18T21:51:57.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.556128+0000", 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.556130+0000", 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4099999999999998e-06 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.556153+0000", 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3464000000000001e-05 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.556188+0000", 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4258999999999998e-05 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.556212+0000", 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.459e-05 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.556424+0000", 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021148200000000001 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.556453+0000", 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9343e-05 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:51:57.560 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 1 2024-04-18T21:51:57.561 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:57.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:51:57.593 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:51:57.593 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:51:57.609 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:57.607+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:57.623 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:51:57.624 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:51:57.710 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:57.707+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:57.726 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:57.726 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:51:57.726 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:51:57.726 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:51:57.726 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:57.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:51:57.804 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:51:57.805 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:51:57.805 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:51:57.806 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:51:57.810 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:57.807+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:57.911 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:57.907+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:57.972 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:57.967+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:57.976 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:57.976 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:82 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050538+0000", 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.923067119000001, 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18845704499999999, 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.977 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 82 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050538+0000", 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050538+0000", 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050539+0000", 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.641e-06 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050791+0000", 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000251776 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050804+0000", 2024-04-18T21:51:57.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2969000000000001e-05 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050811+0000", 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8340000000000001e-06 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051019+0000", 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020797 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051129+0000", 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000110055 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.051931+0000", 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00080212199999999995 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138340+0000", 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.086409092000000007 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.238894+0000", 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100553727 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.238995+0000", 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100859 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.982 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:83 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.922780081999999, 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28825372900000001, 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 83 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2e-07 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050828+0000", 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3929999999999998e-06 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050832+0000", 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4909999999999999e-06 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050836+0000", 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5219999999999999e-06 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052455+0000", 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016189710000000001 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052552+0000", 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7213999999999995e-05 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052983+0000", 2024-04-18T21:51:57.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043119300000000002 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138342+0000", 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.085359090999999998 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.338942+0000", 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200600308 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339078+0000", 2024-04-18T21:51:57.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136026 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:84 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.922751423999999, 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59192865299999997, 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.988 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 84 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:51:57.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050854+0000", 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1399999999999998e-07 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050855+0000", 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5719999999999999e-06 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050860+0000", 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050864+0000", 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5750000000000001e-06 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138083+0000", 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087218981000000001 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138226+0000", 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000143545 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339757+0000", 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20153072 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339856+0000", 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9247999999999996e-05 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642782+0000", 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30292579400000003 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 30/26 ECSubWriteReply(tid=8, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.053696+0000", 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.919908602, 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.184830571, 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.993 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.053696+0000", 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.053696+0000", 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.053697+0000", 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6000000000000004e-07 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.053701+0000", 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8369999999999999e-06 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.053732+0000", 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0948999999999998e-05 2024-04-18T21:51:57.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.053739+0000", 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6580000000000002e-06 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.238482+0000", 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18474323300000001 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.238527+0000", 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4333999999999999e-05 2024-04-18T21:51:57.996 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 30/26 ECSubWriteReply(tid=7, last_complete=30'5, committed=1, applied=1))", 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.055108+0000", 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.918496498, 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18363890399999999, 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055108+0000", 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055108+0000", 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055109+0000", 2024-04-18T21:51:57.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7400000000000002e-07 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055118+0000", 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8629999999999997e-06 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055138+0000", 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0441e-05 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:57.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055146+0000", 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1280000000000008e-06 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.238735+0000", 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18358808400000001 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.238747+0000", 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2614e-05 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.000 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 30/26 ECSubWriteReply(tid=8, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.055694+0000", 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.917910586, 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.183355936, 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055694+0000", 2024-04-18T21:51:58.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055694+0000", 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055695+0000", 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4499999999999998e-07 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055697+0000", 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4099999999999998e-06 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055707+0000", 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0349999999999999e-05 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.055714+0000", 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8739999999999997e-06 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.239039+0000", 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.183324398 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.239050+0000", 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1559000000000001e-05 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 30/26 ECSubWriteReply(tid=8, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.155051+0000", 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.818553759, 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.184546987, 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.155051+0000", 2024-04-18T21:51:58.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.155051+0000", 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.155052+0000", 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0869999999999999e-06 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.155059+0000", 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6769999999999998e-06 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.155076+0000", 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7708999999999999e-05 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.155088+0000", 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1240000000000001e-05 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.338827+0000", 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.183739718 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339598+0000", 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00077055600000000004 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.es0 30/26 ECSubRead(tid=10, to_read={3:77cf8780:::smithi09731216-28:head=0,4096,0}, subchunks={3:77cf8780:::smithi09731216-28:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:51:58.008 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.633590850999999, 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218228800000002, 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:51:58.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340093+0000", 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8665000000000007e-05 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440291+0000", 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100198545 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641845+0000", 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20155352000000001 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642196+0000", 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035155800000000002 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 30/26 ECSubWriteReply(tid=9, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.632908727, 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20052165999999999, 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.012 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:51:58.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340697+0000", 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.004e-06 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340703+0000", 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5260000000000001e-06 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340719+0000", 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.679e-05 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340729+0000", 2024-04-18T21:51:58.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9429999999999995e-06 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541183+0000", 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200453357 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541218+0000", 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5040000000000003e-05 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340933+0000", 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.632672161, 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200445823, 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340933+0000", 2024-04-18T21:51:58.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340933+0000", 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340933+0000", 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.61e-07 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340943+0000", 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4599999999999992e-06 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340953+0000", 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.013e-05 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340958+0000", 2024-04-18T21:51:58.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2939999999999996e-06 2024-04-18T21:51:58.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541310+0000", 2024-04-18T21:51:58.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20035136200000001 2024-04-18T21:51:58.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541378+0000", 2024-04-18T21:51:58.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8816e-05 2024-04-18T21:51:58.021 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.022 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.022 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.022 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.532750667, 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20088825699999999, 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440856+0000", 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9870000000000002e-06 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440895+0000", 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9325e-05 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440929+0000", 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3293000000000001e-05 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440950+0000", 2024-04-18T21:51:58.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0812999999999999e-05 2024-04-18T21:51:58.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641679+0000", 2024-04-18T21:51:58.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20072974199999999 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641742+0000", 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3096999999999996e-05 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.027 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.532392781, 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20123138300000001, 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441213+0000", 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.057e-06 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441231+0000", 2024-04-18T21:51:58.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7995999999999998e-05 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441251+0000", 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0032999999999999e-05 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441267+0000", 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5614e-05 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642333+0000", 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201065943 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642443+0000", 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011074 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 30/26 ECSubWriteReply(tid=9, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:51:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.441513+0000", 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.532092167, 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200115188, 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441513+0000", 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441513+0000", 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441514+0000", 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1939999999999999e-06 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441521+0000", 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0400000000000004e-06 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441540+0000", 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9296e-05 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441556+0000", 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6160000000000001e-05 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641561+0000", 2024-04-18T21:51:58.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20000506100000001 2024-04-18T21:51:58.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641628+0000", 2024-04-18T21:51:58.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6437000000000001e-05 2024-04-18T21:51:58.035 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.cs7 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.210049672, 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30181939000000002, 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763557+0000", 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.776e-06 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763600+0000", 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3120000000000001e-05 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763635+0000", 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5199e-05 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.863748+0000", 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100112399 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964004+0000", 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10025664500000001 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964030+0000", 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5299999999999998e-05 2024-04-18T21:51:58.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065244+0000", 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101214472 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065374+0000", 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000130479 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=19, last_complete=35'11, committed=1, applied=1))", 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6953939150000004, 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22495391000000001, 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:51:58.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278212+0000", 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.004e-06 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278218+0000", 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7490000000000004e-06 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278233+0000", 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5815999999999999e-05 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378308+0000", 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007474600000001 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.502895+0000", 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.124586697 2024-04-18T21:51:58.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.503165+0000", 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026989800000000002 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=20, last_complete=35'12, committed=1, applied=1))", 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.5952678819999999, 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225695955, 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.045 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378338+0000", 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3499999999999996e-07 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378343+0000", 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0669999999999999e-06 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378356+0000", 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3264e-05 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378367+0000", 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1069e-05 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.603689+0000", 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22532233900000001 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604033+0000", 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034358099999999998 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.048 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=21, last_complete=35'13, committed=1, applied=1))", 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.5952275270000005, 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22690856100000001, 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:51:58.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378378+0000", 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378380+0000", 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7190000000000001e-06 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378389+0000", 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378395+0000", 2024-04-18T21:51:58.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5210000000000002e-06 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604185+0000", 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22579012100000001 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.605286+0000", 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011008979999999999 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 3.18s2,from:0'0,to:34'10,epoch:35/31,start:3:18000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378528+0000", 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.5950769940000002, 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20032903399999999, 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378528+0000", 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378528+0000", 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378528+0000", 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2899999999999999e-07 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378533+0000", 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6229999999999996e-06 2024-04-18T21:51:58.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378542+0000", 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.329e-06 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.478630+0000", 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10008789999999999 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.578810+0000", 2024-04-18T21:51:58.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100180509 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.578825+0000", 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4481000000000001e-05 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.578857+0000", 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1863000000000003e-05 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:230 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.9603361780000004, 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.312981385, 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 230 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013270+0000", 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5300000000000002e-07 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013275+0000", 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013291+0000", 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5466000000000001e-05 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013303+0000", 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2e-05 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013750+0000", 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044700300000000002 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013767+0000", 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7263999999999999e-05 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225043+0000", 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21127590299999999 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225199+0000", 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155716 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.326250+0000", 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10105134 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.061 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.bs3 36/26 ECSubWrite(tid=36, reqid=client.4509.0:216, at_version=36'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.9552461330000002, 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20158620099999999, 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1299999999999999e-07 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018417+0000", 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7250000000000002e-05 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018446+0000", 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9567e-05 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118544+0000", 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10009759999999999 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118768+0000", 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022446200000000001 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118785+0000", 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6286e-05 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.219890+0000", 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10110504400000001 2024-04-18T21:51:58.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.219945+0000", 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5278999999999997e-05 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.066 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:58.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:51:58.068 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:51:58.068 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.2 to 1 (10000) 2024-04-18T21:51:58.069 INFO:tasks.rados.rados.0.smithi097.stdout:115: done (6 left) 2024-04-18T21:51:58.069 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 9 current snap is 4 2024-04-18T21:51:58.069 INFO:tasks.rados.rados.0.smithi097.stdout:116: seq_num 65 ranges {147456=720896,868352=376832} 2024-04-18T21:51:58.070 INFO:tasks.rados.rados.0.smithi097.stdout:116: writing smithi09731216-9 from 147456 to 868352 tid 1 2024-04-18T21:51:58.070 INFO:tasks.rados.rados.0.smithi097.stdout:116: writing smithi09731216-9 from 868352 to 1245184 tid 2 2024-04-18T21:51:58.070 INFO:tasks.rados.rados.0.smithi097.stdout:117: rollback oid 41 current snap is 4 2024-04-18T21:51:58.070 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 41 to 0 2024-04-18T21:51:58.070 INFO:tasks.rados.rados.0.smithi097.stdout:108: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-18T21:51:58.070 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:51:58.067+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:51:58.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-18T21:51:58.085 INFO:tasks.rados.rados.0.smithi097.stdout:113: finishing copy_from racing read to smithi09731216-39 2024-04-18T21:51:58.085 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 3 v 49 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-04-18T21:51:58.085 INFO:tasks.rados.rados.0.smithi097.stdout:113: finishing copy_from to smithi09731216-39 2024-04-18T21:51:58.085 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 39 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-18T21:51:58.085 INFO:tasks.rados.rados.0.smithi097.stdout:114: finishing write tid 1 to smithi09731216-18 2024-04-18T21:51:58.085 INFO:tasks.rados.rados.0.smithi097.stdout:114: finishing write tid 2 to smithi09731216-18 2024-04-18T21:51:58.086 INFO:tasks.rados.rados.0.smithi097.stdout:114: finishing write tid 3 to smithi09731216-18 2024-04-18T21:51:58.086 INFO:tasks.rados.rados.0.smithi097.stdout:108: done (7 left) 2024-04-18T21:51:58.086 INFO:tasks.rados.rados.0.smithi097.stdout:111: done (6 left) 2024-04-18T21:51:58.086 INFO:tasks.rados.rados.0.smithi097.stdout:113: done (5 left) 2024-04-18T21:51:58.086 INFO:tasks.rados.rados.0.smithi097.stdout:118: read oid 3 snap -1 2024-04-18T21:51:58.086 INFO:tasks.rados.rados.0.smithi097.stdout:118: expect (ObjNum 54 snap 3 seq_num 54) 2024-04-18T21:51:58.086 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 7 current snap is 4 2024-04-18T21:51:58.086 INFO:tasks.rados.rados.0.smithi097.stdout:119: seq_num 66 ranges {0=737280,737280=376832} 2024-04-18T21:51:58.089 INFO:tasks.rados.rados.0.smithi097.stdout:119: writing smithi09731216-7 from 0 to 737280 tid 1 2024-04-18T21:51:58.091 INFO:tasks.rados.rados.0.smithi097.stdout:119: writing smithi09731216-7 from 737280 to 1114112 tid 2 2024-04-18T21:51:58.091 INFO:tasks.rados.rados.0.smithi097.stdout:120: read oid 1 snap 0 2024-04-18T21:51:58.092 INFO:tasks.rados.rados.0.smithi097.stdout:120: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:51:58.092 INFO:tasks.rados.rados.0.smithi097.stdout:121: rollback oid 12 current snap is 4 2024-04-18T21:51:58.092 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 12 to 0 2024-04-18T21:51:58.092 INFO:tasks.rados.rados.0.smithi097.stdout:122: delete oid 23 current snap is 4 2024-04-18T21:51:58.133 INFO:tasks.rados.rados.0.smithi097.stdout:114: finishing write tid 4 to smithi09731216-18 2024-04-18T21:51:58.133 INFO:tasks.rados.rados.0.smithi097.stdout:114: oid 18 updating version 0 to 19 2024-04-18T21:51:58.133 INFO:tasks.rados.rados.0.smithi097.stdout:114: oid 18 updating version 19 to 20 2024-04-18T21:51:58.133 INFO:tasks.rados.rados.0.smithi097.stdout:114: oid 18 updating version 20 to 21 2024-04-18T21:51:58.134 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 18 v 21 (ObjNum 64 snap 4 seq_num 64) dirty exists 2024-04-18T21:51:58.134 INFO:tasks.rados.rados.0.smithi097.stdout:114: left oid 18 (ObjNum 64 snap 4 seq_num 64) 2024-04-18T21:51:58.168 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:58.167+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:58.195 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.9s0 41/31 ECSubReadReply(tid=36, attrs_read=0))", 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.193529+0000", 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.002573217, 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025877909999999999, 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193529+0000", 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193529+0000", 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193530+0000", 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.27e-07 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193546+0000", 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5772000000000001e-05 2024-04-18T21:51:58.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193555+0000", 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5880000000000004e-06 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193562+0000", 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.199e-06 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.198 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:312 3.9s0 3:917033a1:::smithi09731216-1:2 [read 0~32768,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092703+0000", 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.10339943, 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.103451415, 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 312 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092703+0000", 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092703+0000", 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092705+0000", 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.278e-06 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092716+0000", 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0849000000000001e-05 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092736+0000", 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9862999999999999e-05 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092745+0000", 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7730000000000005e-06 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.168270+0000", 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.075524892999999996 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.168448+0000", 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017801199999999999 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:313 3.9s0 3:917033a1:::smithi09731216-1:2 [read 0~32768,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092763+0000", 2024-04-18T21:51:58.202 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.10333958, 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.103432092, 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 313 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092763+0000", 2024-04-18T21:51:58.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092763+0000", 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092763+0000", 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3700000000000001e-07 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092769+0000", 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2379999999999997e-06 2024-04-18T21:51:58.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092773+0000", 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5859999999999998e-06 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.354e-06 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169193+0000", 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.076416365 2024-04-18T21:51:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169295+0000", 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000101677 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:314 3.9s0 3.85cc0e89 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092782+0000", 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.103321049, 2024-04-18T21:51:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.103447675, 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 314 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092782+0000", 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092782+0000", 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092782+0000", 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6399999999999998e-07 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092783+0000", 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.655e-06 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092787+0000", 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6509999999999999e-06 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192857+0000", 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100070254 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.9s0 41/31 ECSubRead(tid=35, to_read={3:917033a1:::smithi09731216-1:head=0,8192,0}, subchunks={3:917033a1:::smithi09731216-1:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.168883+0000", 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.027219759999999999, 2024-04-18T21:51:58.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.027406402, 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.168883+0000", 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.168883+0000", 2024-04-18T21:51:58.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.168883+0000", 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.168883+0000", 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169025+0000", 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014240800000000001 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169036+0000", 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.061e-05 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.9s0 41/31 ECSubRead(tid=36, to_read={3:917033a1:::smithi09731216-1:head=0,8192,0}, subchunks={3:917033a1:::smithi09731216-1:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.02659164, 2024-04-18T21:51:58.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026806416, 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "no flag points reached", 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169540+0000", 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8588000000000001e-05 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.9s0 41/31 ECSubReadReply(tid=35, attrs_read=0))", 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.170325+0000", 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.025777573000000002, 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026013036999999999, 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.215 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170325+0000", 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170325+0000", 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170326+0000", 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9899999999999999e-07 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170366+0000", 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9826999999999997e-05 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170380+0000", 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4014e-05 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170387+0000", 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.922e-06 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.9s0 41/31 ECSubReadReply(tid=36, attrs_read=0))", 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.170614+0000", 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.025488446000000001, 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025747071, 2024-04-18T21:51:58.218 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170614+0000", 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170614+0000", 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170615+0000", 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4199999999999995e-07 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170628+0000", 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3074000000000001e-05 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170640+0000", 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2384e-05 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170647+0000", 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6290000000000004e-06 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.9s0 41/31 ECSubReadReply(tid=35, attrs_read=0))", 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.170683+0000", 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.025419718000000001, 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025701425999999999, 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.221 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170683+0000", 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170683+0000", 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170684+0000", 2024-04-18T21:51:58.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4900000000000005e-07 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170707+0000", 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3777999999999998e-05 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170720+0000", 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2320999999999999e-05 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.170726+0000", 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5570000000000001e-06 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.9s0 41/31 ECSubReadReply(tid=36, attrs_read=0))", 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.171073+0000", 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.025029813000000001, 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025336471999999999, 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.224 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.171073+0000", 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.171073+0000", 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.171073+0000", 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7999999999999995e-07 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.171110+0000", 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7070000000000003e-05 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.171120+0000", 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1929999999999999e-06 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.171126+0000", 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7059999999999998e-06 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:318 3.19s0 3.78f5c5b9 (undecoded) ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0032297659999999998, 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035612180000000001, 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.227 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 318 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192874+0000", 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2699999999999998e-07 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192878+0000", 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0239999999999999e-06 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192886+0000", 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2549999999999997e-06 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192890+0000", 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.357e-06 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.9s0 41/31 ECSubReadReply(tid=35, attrs_read=0))", 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.193461+0000", 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0026418470000000001, 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0030000220000000002, 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.230 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193461+0000", 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193461+0000", 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193461+0000", 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3499999999999995e-07 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193491+0000", 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9819e-05 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193504+0000", 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2824e-05 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.193511+0000", 2024-04-18T21:51:58.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3119999999999996e-06 2024-04-18T21:51:58.233 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.233 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.233 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.233 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.233 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:51:58.233 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 12 2024-04-18T21:51:58.233 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:58.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:51:58.269 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:51:58.267+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:51:58.284 INFO:tasks.rados.rados.0.smithi097.stdout:99: finishing write tid 1 to smithi09731216-47 2024-04-18T21:51:58.285 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:58.283+0000 7f2be1fa5640 -1 log_channel(cluster) log [ERR] : osd.1 pg[3.18s7]: reservation requested while still reserved 2024-04-18T21:51:58.369 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:58.367+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:58.388 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:58.388 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:51:58.388 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:51:58.388 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:51:58.388 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:58.390 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:58.387+0000 7f873c9bb640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3 to read 0,1,2,3,4,5,6,7 2024-04-18T21:51:58.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:51:58.470 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:51:58.471+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:51:58.476 INFO:tasks.rados.rados.0.smithi097.stdout:120: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:51:58.480 INFO:tasks.rados.rados.0.smithi097.stdout:116: finishing write tid 1 to smithi09731216-9 2024-04-18T21:51:58.493 INFO:tasks.rados.rados.0.smithi097.stdout:117: finishing rollback tid 0 to smithi09731216-41 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 30/26 ECSubWriteReply(tid=9, last_complete=29'6, committed=1, applied=1))", 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.348744+0000", 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 12.212290092, 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201938637, 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.348744+0000", 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.348744+0000", 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.348745+0000", 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.897e-06 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.348757+0000", 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1753000000000001e-05 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.348789+0000", 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1946999999999998e-05 2024-04-18T21:51:58.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.448895+0000", 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100105603 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.550635+0000", 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101740021 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.550682+0000", 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7416000000000001e-05 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:103 3.8s0 3:1b279e7b:::smithi09731216-35:head [append 0~573440,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.663348+0000", 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.89768572, 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.405152289, 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.568 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 103 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.663348+0000", 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.663348+0000", 2024-04-18T21:51:58.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.663351+0000", 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1429999999999998e-06 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.664055+0000", 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00070422499999999995 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.664090+0000", 2024-04-18T21:51:58.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4428999999999999e-05 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764202+0000", 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10011234300000001 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764380+0000", 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017786899999999999 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764755+0000", 2024-04-18T21:51:58.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037547399999999999 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.767507+0000", 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027519369999999999 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866232+0000", 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098724751999999999 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.068298+0000", 2024-04-18T21:51:58.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.202066089 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.068500+0000", 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020202800000000001 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:104 3.8s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.764264+0000", 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.796769403000001, 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31182509600000002, 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.573 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764264+0000", 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764264+0000", 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764267+0000", 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5660000000000001e-06 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764282+0000", 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5116999999999999e-05 2024-04-18T21:51:58.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764311+0000", 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8765999999999999e-05 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764331+0000", 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9797e-05 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.864577+0000", 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100246944 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.864853+0000", 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027538200000000001 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:58.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.865734+0000", 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00088124700000000004 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866586+0000", 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00085225000000000003 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.075663+0000", 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20907698999999999 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.076089+0000", 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00042603699999999999 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.8s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.796674296999999, 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51395767000000003, 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.579 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764361+0000", 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.61e-06 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764368+0000", 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1600000000000001e-06 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764389+0000", 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1073e-05 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764409+0000", 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0006000000000001e-05 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866089+0000", 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101679558 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866221+0000", 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132022 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278264+0000", 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41204366100000001 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278317+0000", 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2580000000000001e-05 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.6s1 30/26 ECSubWrite(tid=8, reqid=client.4509.0:107, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:58.583 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.782957+0000", 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.778076266999999, 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20204204200000001, 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.782957+0000", 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.782957+0000", 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.782959+0000", 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.66e-06 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.782991+0000", 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1707e-05 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.783023+0000", 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2209000000000003e-05 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.883129+0000", 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100106003 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.983522+0000", 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10039279399999999 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.983551+0000", 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9696000000000002e-05 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.984830+0000", 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001278581 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.984999+0000", 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016939200000000001 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 30/26 ECSubWriteReply(tid=12, last_complete=30'2, committed=1, applied=1))", 2024-04-18T21:51:58.587 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.967105+0000", 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.593928751, 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20572794799999999, 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967105+0000", 2024-04-18T21:51:58.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967105+0000", 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967106+0000", 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1299999999999998e-07 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967112+0000", 2024-04-18T21:51:58.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8000000000000004e-06 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967130+0000", 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8257000000000002e-05 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.067226+0000", 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100096033 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.172781+0000", 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105555522 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.172833+0000", 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1422999999999999e-05 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:103 3.18s0 3:1b279e7b:::smithi09731216-35:head [append 0~573440,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:51:58.591 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.580015833000001, 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6106588159999999, 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 103 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981021+0000", 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5470000000000001e-06 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981778+0000", 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00075675099999999995 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981811+0000", 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3200000000000001e-05 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981832+0000", 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0703000000000002e-05 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.277997+0000", 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29616472700000002 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278003+0000", 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3269999999999997e-06 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591581+0000", 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.313578111 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591677+0000", 2024-04-18T21:51:58.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5450000000000003e-05 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:104 3.18s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.579150979, 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714877378, 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.596 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981885+0000", 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0260000000000002e-06 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981895+0000", 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7969999999999995e-06 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981918+0000", 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3116e-05 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981931+0000", 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2850000000000001e-05 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278086+0000", 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29615502199999999 2024-04-18T21:51:58.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278088+0000", 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6230000000000001e-06 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696611+0000", 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418522515 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696760+0000", 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149429 2024-04-18T21:51:58.600 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.579086673000001, 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9142408849999999, 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.601 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981948+0000", 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981952+0000", 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8739999999999997e-06 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981963+0000", 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.133e-05 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981971+0000", 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2900000000000002e-06 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278161+0000", 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29618945299999999 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278163+0000", 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.734e-06 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696819+0000", 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418656006 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.697058+0000", 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023910700000000001 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.896188+0000", 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19913024800000001 2024-04-18T21:51:58.605 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.19s0 35/31 ECSubRead(tid=27, to_read={3:9da3af1e:::smithi09731216-23:head=0,45056,0}, subchunks={3:9da3af1e:::smithi09731216-23:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.278940+0000", 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 6.2820939669999998, 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20108193499999999, 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278940+0000", 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278940+0000", 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278940+0000", 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278940+0000", 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278956+0000", 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5812e-05 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.379033+0000", 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100077078 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.479247+0000", 2024-04-18T21:51:58.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100214188 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.480022+0000", 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000774857 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:225 3.19s0 3:9941efaf:::smithi09731216-5:head [read 0~81920,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.012932+0000", 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.5481017320000001, 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40503983500000001, 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 225 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.610 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012932+0000", 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012932+0000", 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012935+0000", 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0199999999999999e-06 2024-04-18T21:51:58.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012954+0000", 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9133000000000001e-05 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012995+0000", 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0694e-05 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013019+0000", 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4142999999999999e-05 2024-04-18T21:51:58.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013185+0000", 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016607199999999999 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013204+0000", 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8828999999999999e-05 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.212761+0000", 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19955726600000001 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.212899+0000", 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013827199999999999 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.417972+0000", 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20507240600000001 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:226 3.19s0 3:9941efaf:::smithi09731216-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013083+0000", 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.547950363, 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40562686100000001, 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 226 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013083+0000", 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013083+0000", 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013085+0000", 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.53e-06 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013095+0000", 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0319999999999999e-05 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013118+0000", 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2847e-05 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013136+0000", 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7535999999999999e-05 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013367+0000", 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000231375 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013389+0000", 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1946999999999998e-05 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.213788+0000", 2024-04-18T21:51:58.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20039871200000001 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.214027+0000", 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023972899999999999 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.418710+0000", 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20468286499999999 2024-04-18T21:51:58.619 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:227 3.19s0 3:9941efaf:::smithi09731216-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.5478743929999998, 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50662614500000003, 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.620 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 227 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013160+0000", 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1299999999999999e-07 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013167+0000", 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9670000000000004e-06 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013187+0000", 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9726000000000001e-05 2024-04-18T21:51:58.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013202+0000", 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5483000000000001e-05 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013460+0000", 2024-04-18T21:51:58.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025794499999999998 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013468+0000", 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8460000000000008e-06 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.315837+0000", 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30236893999999997 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.316131+0000", 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029429400000000001 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.519785+0000", 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20365413099999999 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=61, reqid=client.4509.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=36'25))", 2024-04-18T21:51:58.625 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.707673+0000", 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.8533602590000005, 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20204561700000001, 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.707673+0000", 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.707673+0000", 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.707675+0000", 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7859999999999999e-06 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.707698+0000", 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.268e-05 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.707732+0000", 2024-04-18T21:51:58.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4338999999999998e-05 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.707750+0000", 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8057e-05 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.808012+0000", 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100261207 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.808026+0000", 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4389999999999999e-05 2024-04-18T21:51:58.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.909644+0000", 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101617719 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.909719+0000", 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5438999999999994e-05 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.3s1 37/31 ECSubWrite(tid=63, reqid=osd.4.0:63, at_version=37'27, trim_to=0'0, roll_forward_to=36'25))", 2024-04-18T21:51:58.629 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:54.113205+0000", 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.4478285260000003, 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20158675500000001, 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.113205+0000", 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.113205+0000", 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.113206+0000", 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0699999999999998e-07 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.113233+0000", 2024-04-18T21:51:58.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7274e-05 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.113247+0000", 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4219000000000001e-05 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.113259+0000", 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1703e-05 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.213401+0000", 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10014266099999999 2024-04-18T21:51:58.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.213411+0000", 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7780000000000002e-06 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.314754+0000", 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10134315300000001 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:54.314792+0000", 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.756e-05 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.3s1 40/31 ECSubWrite(tid=90, reqid=client.4509.0:289, at_version=0'0, trim_to=0'0, roll_forward_to=39'49))", 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.454345+0000", 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.106688203, 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20134492200000001, 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.634 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.454345+0000", 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.454345+0000", 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.454347+0000", 2024-04-18T21:51:58.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1489999999999999e-06 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.454361+0000", 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4565e-05 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.454380+0000", 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9259e-05 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.454391+0000", 2024-04-18T21:51:58.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0069e-05 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.654665+0000", 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20027479300000001 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.654682+0000", 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6646999999999998e-05 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.655658+0000", 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00097646899999999999 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.655690+0000", 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1971000000000003e-05 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:312 3.9s0 3:917033a1:::smithi09731216-1:2 [read 0~32768,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092703+0000", 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.468330515, 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27879774400000001, 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 312 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.639 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092703+0000", 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092703+0000", 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092705+0000", 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.278e-06 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092716+0000", 2024-04-18T21:51:58.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0849000000000001e-05 2024-04-18T21:51:58.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092736+0000", 2024-04-18T21:51:58.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9862999999999999e-05 2024-04-18T21:51:58.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092745+0000", 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7730000000000005e-06 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.168270+0000", 2024-04-18T21:51:58.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.075524892999999996 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.168448+0000", 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017801199999999999 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.371501+0000", 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203053076 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:313 3.9s0 3:917033a1:::smithi09731216-1:2 [read 0~32768,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092763+0000", 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.46827066499999997, 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27923321899999998, 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 313 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.645 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092763+0000", 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092763+0000", 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092763+0000", 2024-04-18T21:51:58.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3700000000000001e-07 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092769+0000", 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2379999999999997e-06 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092773+0000", 2024-04-18T21:51:58.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5859999999999998e-06 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.354e-06 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169193+0000", 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.076416365 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169295+0000", 2024-04-18T21:51:58.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000101677 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.371996+0000", 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.202701662 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:314 3.9s0 3:917033a1:::smithi09731216-1:2 [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092782+0000", 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.46825213399999999, 2024-04-18T21:51:58.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38207651599999998, 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 314 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092782+0000", 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092782+0000", 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092782+0000", 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6399999999999998e-07 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092783+0000", 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.655e-06 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092787+0000", 2024-04-18T21:51:58.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6509999999999999e-06 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192857+0000", 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100070254 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.371529+0000", 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17867122699999999 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.371597+0000", 2024-04-18T21:51:58.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8230000000000002e-05 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.474858+0000", 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10326123500000001 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.9s0 41/31 ECSubRead(tid=36, to_read={3:917033a1:::smithi09731216-1:head=0,8192,0}, subchunks={3:917033a1:::smithi09731216-1:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:58.654 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.39152272500000002, 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20182382900000001, 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:51:58.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169511+0000", 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:51:58.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.169540+0000", 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8588000000000001e-05 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.269620+0000", 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100080788 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.371277+0000", 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10165679399999999 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:51:58.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.371335+0000", 2024-04-18T21:51:58.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7658999999999998e-05 2024-04-18T21:51:58.658 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.658 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.658 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.658 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:51:58.658 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:51:58.658 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:51:58.659 INFO:tasks.rados.rados.0.smithi097.stdout:117: finishing rollback tid 1 to smithi09731216-41 2024-04-18T21:51:58.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:51:58.660 INFO:tasks.rados.rados.0.smithi097.stdout:116: finishing write tid 2 to smithi09731216-9 2024-04-18T21:51:58.660 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:58.655+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:51:58.667 INFO:tasks.rados.rados.0.smithi097.stdout:117: finishing rollback tid 2 to smithi09731216-41 2024-04-18T21:51:58.667 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 41 v 12 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-18T21:51:58.706 INFO:tasks.rados.rados.0.smithi097.stdout:99: finishing write tid 2 to smithi09731216-47 2024-04-18T21:51:58.707 INFO:tasks.rados.rados.0.smithi097.stdout:99: finishing write tid 3 to smithi09731216-47 2024-04-18T21:51:58.760 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:58.759+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:58.780 INFO:tasks.rados.rados.0.smithi097.stdout:116: finishing write tid 3 to smithi09731216-9 2024-04-18T21:51:58.783 INFO:tasks.rados.rados.0.smithi097.stdout:119: finishing write tid 1 to smithi09731216-7 2024-04-18T21:51:58.783 INFO:tasks.rados.rados.0.smithi097.stdout:119: finishing write tid 2 to smithi09731216-7 2024-04-18T21:51:58.784 INFO:tasks.rados.rados.0.smithi097.stdout:119: finishing write tid 3 to smithi09731216-7 2024-04-18T21:51:58.795 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:58.796 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:58.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.796 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 41/31 ECSubReadReply(tid=93, attrs_read=0))", 2024-04-18T21:51:58.796 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.695615+0000", 2024-04-18T21:51:58.796 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.099828495000000003, 2024-04-18T21:51:58.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099865568000000002, 2024-04-18T21:51:58.797 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:58.797 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:58.797 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.797 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:58.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:58.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.695615+0000", 2024-04-18T21:51:58.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.695615+0000", 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.695616+0000", 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0399999999999996e-07 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.695888+0000", 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027206000000000002 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.695899+0000", 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1749000000000001e-05 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.695906+0000", 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0269999999999998e-06 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.800 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 41/31 ECSubReadReply(tid=92, attrs_read=0))", 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.696036+0000", 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.099406962000000001, 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099530572999999997, 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696036+0000", 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696036+0000", 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:58.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696037+0000", 2024-04-18T21:51:58.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6700000000000003e-07 2024-04-18T21:51:58.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:58.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696270+0000", 2024-04-18T21:51:58.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023316100000000001 2024-04-18T21:51:58.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:58.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696281+0000", 2024-04-18T21:51:58.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1022000000000001e-05 2024-04-18T21:51:58.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696287+0000", 2024-04-18T21:51:58.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8069999999999996e-06 2024-04-18T21:51:58.805 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.806 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:58.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.806 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.19s6 41/31 ECSubWrite(tid=38, reqid=client.4509.0:318, at_version=41'9, trim_to=0'0, roll_forward_to=33'7))", 2024-04-18T21:51:58.806 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.696322+0000", 2024-04-18T21:51:58.807 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.099121870000000001, 2024-04-18T21:51:58.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099325188999999994, 2024-04-18T21:51:58.807 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:58.807 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:58.807 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.807 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:58.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:58.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696322+0000", 2024-04-18T21:51:58.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:58.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696322+0000", 2024-04-18T21:51:58.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:58.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696322+0000", 2024-04-18T21:51:58.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9499999999999998e-07 2024-04-18T21:51:58.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:58.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696355+0000", 2024-04-18T21:51:58.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3148999999999997e-05 2024-04-18T21:51:58.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:58.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696366+0000", 2024-04-18T21:51:58.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0757999999999999e-05 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696377+0000", 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1524e-05 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:306 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.70911056400000005, 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70941109800000002, 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 306 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.813 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:58.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086334+0000", 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4900000000000004e-07 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086340+0000", 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0900000000000001e-06 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086359+0000", 2024-04-18T21:51:58.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8940000000000002e-05 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086369+0000", 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7569999999999999e-06 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086608+0000", 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000239702 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086634+0000", 2024-04-18T21:51:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5948999999999998e-05 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.291884+0000", 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20524995300000001 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292051+0000", 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000166797 2024-04-18T21:51:58.817 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 41/31 ECSubReadReply(tid=93, attrs_read=0))", 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.696484+0000", 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.098959468999999994, 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099384917000000003, 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:58.818 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696484+0000", 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:58.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696484+0000", 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696484+0000", 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1600000000000001e-07 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696764+0000", 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027913100000000001 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696789+0000", 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5440999999999999e-05 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696799+0000", 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.025e-05 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.821 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:307 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.709060476, 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70956622599999997, 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 307 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.822 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086384+0000", 2024-04-18T21:51:58.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9699999999999995e-07 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086388+0000", 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3900000000000003e-06 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086398+0000", 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8369999999999992e-06 2024-04-18T21:51:58.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086406+0000", 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3599999999999996e-06 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086727+0000", 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032101900000000002 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086748+0000", 2024-04-18T21:51:58.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0917e-05 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292674+0000", 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20592602600000001 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:58.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292826+0000", 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015169700000000001 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.10s5 41/39 ECSubWrite(tid=6, reqid=client.4509.0:267, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.705372+0000", 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.090071546000000002, 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.090699786000000004, 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "no flag points reached", 2024-04-18T21:51:58.827 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.705372+0000", 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.705372+0000", 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.705373+0000", 2024-04-18T21:51:58.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2920000000000001e-06 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.705642+0000", 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026847600000000002 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.705660+0000", 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8085e-05 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.829 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:58.830 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.830 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:58.830 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:51:58.830 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 7 2024-04-18T21:51:58.830 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:58.831 INFO:tasks.rados.rados.0.smithi097.stdout:121: finishing rollback tid 0 to smithi09731216-12 2024-04-18T21:51:58.831 INFO:tasks.rados.rados.0.smithi097.stdout:121: finishing rollback tid 1 to smithi09731216-12 2024-04-18T21:51:58.832 INFO:tasks.rados.rados.0.smithi097.stdout:121: finishing rollback tid 2 to smithi09731216-12 2024-04-18T21:51:58.832 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 12 v 19 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-18T21:51:58.832 INFO:tasks.rados.rados.0.smithi097.stdout:114: done (9 left) 2024-04-18T21:51:58.832 INFO:tasks.rados.rados.0.smithi097.stdout:117: done (8 left) 2024-04-18T21:51:58.832 INFO:tasks.rados.rados.0.smithi097.stdout:120: done (7 left) 2024-04-18T21:51:58.832 INFO:tasks.rados.rados.0.smithi097.stdout:121: done (6 left) 2024-04-18T21:51:58.832 INFO:tasks.rados.rados.0.smithi097.stdout:122: done (5 left) 2024-04-18T21:51:58.832 INFO:tasks.rados.rados.0.smithi097.stdout:123: delete oid 30 current snap is 4 2024-04-18T21:51:58.832 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:51:58.872 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:51:58.871+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:51:58.885 INFO:tasks.rados.rados.0.smithi097.stdout:116: finishing write tid 4 to smithi09731216-9 2024-04-18T21:51:58.885 INFO:tasks.rados.rados.0.smithi097.stdout:116: oid 9 updating version 0 to 23 2024-04-18T21:51:58.885 INFO:tasks.rados.rados.0.smithi097.stdout:116: oid 9 version 23 is already newer than 22 2024-04-18T21:51:58.885 INFO:tasks.rados.rados.0.smithi097.stdout:116: oid 9 version 23 is already newer than 21 2024-04-18T21:51:58.885 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 9 v 23 (ObjNum 65 snap 4 seq_num 65) dirty exists 2024-04-18T21:51:58.885 INFO:tasks.rados.rados.0.smithi097.stdout:116: left oid 9 (ObjNum 65 snap 4 seq_num 65) 2024-04-18T21:51:58.909 INFO:tasks.rados.rados.0.smithi097.stdout:118: expect (ObjNum 54 snap 3 seq_num 54) 2024-04-18T21:51:58.962 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:51:58.959+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:51:58.996 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:58.996 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:58.996 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:58.996 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:58.996 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:59.010 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:51:59.017 INFO:tasks.rados.rados.0.smithi097.stdout:102: finishing write tid 1 to smithi09731216-19 2024-04-18T21:51:59.018 INFO:tasks.rados.rados.0.smithi097.stdout:102: finishing write tid 2 to smithi09731216-19 2024-04-18T21:51:59.018 INFO:tasks.rados.rados.0.smithi097.stdout:102: finishing write tid 3 to smithi09731216-19 2024-04-18T21:51:59.056 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.3 to 1 (10000) 2024-04-18T21:51:59.063 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:59.059+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:59.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-18T21:51:59.088 INFO:tasks.rados.rados.0.smithi097.stdout:119: finishing write tid 4 to smithi09731216-7 2024-04-18T21:51:59.088 INFO:tasks.rados.rados.0.smithi097.stdout:119: oid 7 updating version 0 to 9 2024-04-18T21:51:59.088 INFO:tasks.rados.rados.0.smithi097.stdout:119: oid 7 updating version 9 to 11 2024-04-18T21:51:59.088 INFO:tasks.rados.rados.0.smithi097.stdout:119: oid 7 version 11 is already newer than 10 2024-04-18T21:51:59.088 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 7 v 11 (ObjNum 66 snap 4 seq_num 66) dirty exists 2024-04-18T21:51:59.088 INFO:tasks.rados.rados.0.smithi097.stdout:119: left oid 7 (ObjNum 66 snap 4 seq_num 66) 2024-04-18T21:51:59.104 INFO:tasks.rados.rados.0.smithi097.stdout:116: done (5 left) 2024-04-18T21:51:59.104 INFO:tasks.rados.rados.0.smithi097.stdout:118: done (4 left) 2024-04-18T21:51:59.104 INFO:tasks.rados.rados.0.smithi097.stdout:119: done (3 left) 2024-04-18T21:51:59.104 INFO:tasks.rados.rados.0.smithi097.stdout:123: done (2 left) 2024-04-18T21:51:59.105 INFO:tasks.rados.rados.0.smithi097.stdout:124: copy_from oid 12 from oid 8 current snap is 4 2024-04-18T21:51:59.105 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 38 current snap is 4 2024-04-18T21:51:59.105 INFO:tasks.rados.rados.0.smithi097.stdout:125: seq_num 69 ranges {1589248=786432,2375680=376832} 2024-04-18T21:51:59.121 INFO:tasks.rados.rados.0.smithi097.stdout:125: writing smithi09731216-38 from 1589248 to 2375680 tid 1 2024-04-18T21:51:59.124 INFO:tasks.rados.rados.0.smithi097.stdout:125: writing smithi09731216-38 from 2375680 to 2752512 tid 2 2024-04-18T21:51:59.124 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 35 current snap is 4 2024-04-18T21:51:59.124 INFO:tasks.rados.rados.0.smithi097.stdout:126: seq_num 70 ranges {573440=802816,1376256=376832} 2024-04-18T21:51:59.134 INFO:tasks.rados.rados.0.smithi097.stdout:126: writing smithi09731216-35 from 573440 to 1376256 tid 1 2024-04-18T21:51:59.135 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:59.135 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:59.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:59.135 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:59.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.135 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.877168179, 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82320560300000001, 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254837+0000", 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9000000000000002e-06 2024-04-18T21:51:59.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255465+0000", 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000628435 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255498+0000", 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2305999999999999e-05 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255523+0000", 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5137e-05 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078012+0000", 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82248975599999996 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078039+0000", 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7069000000000001e-05 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.876441273999999, 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82259326700000002, 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.140 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255563+0000", 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.72e-06 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255574+0000", 2024-04-18T21:51:59.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0993e-05 2024-04-18T21:51:59.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255598+0000", 2024-04-18T21:51:59.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4329999999999999e-05 2024-04-18T21:51:59.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255615+0000", 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7569000000000001e-05 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078131+0000", 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82251584300000002 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078154+0000", 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2812000000000001e-05 2024-04-18T21:51:59.144 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:93 3.3s0 3.fef99513 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.876353332000001, 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822627518, 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.145 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 93 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255650+0000", 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2330000000000001e-06 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255657+0000", 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5000000000000002e-06 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255681+0000", 2024-04-18T21:51:59.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3354e-05 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255705+0000", 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3932999999999999e-05 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078240+0000", 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822535609 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078276+0000", 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5889000000000001e-05 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.265591+0000", 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.866411475, 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50576242900000001, 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.149 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265591+0000", 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265591+0000", 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265592+0000", 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2130000000000001e-06 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265611+0000", 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8923000000000002e-05 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265625+0000", 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4763000000000001e-05 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265637+0000", 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1063999999999999e-05 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.771295+0000", 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50565803499999995 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.771353+0000", 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8430999999999997e-05 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.265672+0000", 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.866330396, 2024-04-18T21:51:59.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50577193600000003, 2024-04-18T21:51:59.154 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.154 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.154 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.154 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265672+0000", 2024-04-18T21:51:59.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265672+0000", 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265673+0000", 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.094e-06 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265692+0000", 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9516000000000001e-05 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265716+0000", 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.385e-05 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265733+0000", 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6419999999999999e-05 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.771403+0000", 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50567023200000005 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.771444+0000", 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0824000000000001e-05 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.265921+0000", 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.866080751, 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60568457799999997, 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265921+0000", 2024-04-18T21:51:59.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265921+0000", 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265922+0000", 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1170000000000001e-06 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265938+0000", 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5423000000000001e-05 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265951+0000", 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3617e-05 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.265962+0000", 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0890000000000001e-05 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871553+0000", 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60559044799999995 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871606+0000", 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3083e-05 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s0 29/26 ECSubRead(tid=12, to_read={3:c057a400:::smithi09731216-17:head=0,4096,0}, subchunks={3:c057a400:::smithi09731216-17:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.764129059, 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50402778100000001, 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367873+0000", 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367901+0000", 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8042e-05 2024-04-18T21:51:59.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.367920+0000", 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8536999999999999e-05 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871645+0000", 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50372545800000001 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871901+0000", 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025574400000000002 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.3s0 30/26 ECSubReadReply(tid=12, attrs_read=0))", 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.368470+0000", 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.763531881, 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50362052899999998, 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.166 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368470+0000", 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368470+0000", 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368471+0000", 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8000000000000005e-07 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368481+0000", 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0183000000000001e-05 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.368492+0000", 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0710000000000001e-05 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.468570+0000", 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100077795 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.871983+0000", 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40341337799999999 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.872091+0000", 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010768300000000001 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.169 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs2 30/26 ECSubWrite(tid=11, reqid=client.4509.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.668803+0000", 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.463198951000001, 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40783071599999998, 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668803+0000", 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668803+0000", 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668805+0000", 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.533e-06 2024-04-18T21:51:59.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668819+0000", 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4188e-05 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668842+0000", 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3660999999999999e-05 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.668856+0000", 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3825000000000001e-05 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.873265+0000", 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.204408969 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.873280+0000", 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.45e-05 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.076591+0000", 2024-04-18T21:51:59.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20331132399999999 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.076634+0000", 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2716000000000002e-05 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:73 3.13s0 3:c8da3643:::smithi09731216-25:head [append 0~409600,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:51:59.174 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.935269862, 2024-04-18T21:51:59.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.609707234, 2024-04-18T21:51:59.175 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.175 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.175 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.175 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196734+0000", 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829999999999999e-06 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196977+0000", 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024357200000000001 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196998+0000", 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0503e-05 2024-04-18T21:51:59.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197013+0000", 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5536000000000002e-05 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197060+0000", 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6519e-05 2024-04-18T21:51:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197061+0000", 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2900000000000002e-07 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591261+0000", 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3942002449999999 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591644+0000", 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038315000000000002 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593905+0000", 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022609050000000001 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697736+0000", 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10383110700000001 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806236+0000", 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10850008999999999 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806439+0000", 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020309500000000001 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:74 3.13s0 3:c8da3643:::smithi09731216-25:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.934965518, 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7102205860000002, 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 74 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.182 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197038+0000", 2024-04-18T21:51:59.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9000000000000005e-07 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197043+0000", 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197053+0000", 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0267e-05 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197066+0000", 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.327e-05 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197104+0000", 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8039e-05 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197105+0000", 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8400000000000004e-07 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595282+0000", 2024-04-18T21:51:59.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3981771869999999 2024-04-18T21:51:59.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595543+0000", 2024-04-18T21:51:59.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026070899999999997 2024-04-18T21:51:59.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.596517+0000", 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00097469500000000003 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697956+0000", 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10143811899999999 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907075+0000", 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20911996799999999 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907257+0000", 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018164799999999999 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.13s0 3:c8a99f7f:::smithi09731216-31:head [append 0~507904,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.934439455, 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.812483013, 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:51:59.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8599999999999998e-07 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197879+0000", 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000315495 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197893+0000", 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4143e-05 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197908+0000", 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4932999999999999e-05 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197987+0000", 2024-04-18T21:51:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9288000000000004e-05 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197989+0000", 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.137e-06 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.698727+0000", 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.500737242 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.699015+0000", 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028849499999999999 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.700958+0000", 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019429969999999999 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798717+0000", 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097759288999999999 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.009932+0000", 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.211214651 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010046+0000", 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113657 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.13s0 3:c8a99f7f:::smithi09731216-31:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.934077614, 2024-04-18T21:51:59.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8124102710000001, 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:51:59.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197925+0000", 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000005e-07 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197931+0000", 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6799999999999998e-06 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197943+0000", 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1620000000000001e-05 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197953+0000", 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-05 2024-04-18T21:51:59.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7586000000000001e-05 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5599999999999995e-07 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702431+0000", 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.504419629 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702670+0000", 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023948700000000001 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.703977+0000", 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00130674 2024-04-18T21:51:59.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798719+0000", 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094742397000000006 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010219+0000", 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21149920999999999 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010335+0000", 2024-04-18T21:51:59.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011602 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:231 3.3s0 3:c170a397:::smithi09731216-3:head [append 0~540672] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.026894+0000", 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1051076999999996, 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.479452722, 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.202 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 231 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.026894+0000", 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.026894+0000", 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.026897+0000", 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7599999999999998e-06 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.027835+0000", 2024-04-18T21:51:59.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093806499999999997 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.027867+0000", 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1646000000000002e-05 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.027885+0000", 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8593e-05 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.296617+0000", 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.26873199399999997 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.296742+0000", 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012466800000000001 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.298448+0000", 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001705846 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.301830+0000", 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033818139999999999 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.506211+0000", 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.204381023 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.506347+0000", 2024-04-18T21:51:59.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013631299999999999 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:232 3.3s0 3:c170a397:::smithi09731216-3:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.0989202459999996, 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.67408736800000002, 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.208 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 232 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033085+0000", 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.74e-06 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033688+0000", 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060374300000000001 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033720+0000", 2024-04-18T21:51:59.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2119000000000002e-05 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033737+0000", 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6273000000000002e-05 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.299553+0000", 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.26581622199999999 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.299658+0000", 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105429 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.300593+0000", 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093458899999999999 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.304845+0000", 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042519960000000001 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707032+0000", 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.402187454 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707169+0000", 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013680300000000001 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:233 3.3s0 3:c170a397:::smithi09731216-3:head [setxattr _header (55) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:59.213 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.0982326120000003, 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.67365432700000005, 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 233 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033771+0000", 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2500000000000001e-06 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033776+0000", 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7629999999999997e-06 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033791+0000", 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4803e-05 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033801+0000", 2024-04-18T21:51:59.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6830000000000003e-06 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.302979+0000", 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.26917756799999998 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.303256+0000", 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027797400000000001 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.304206+0000", 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094934700000000002 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.505420+0000", 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201213902 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707355+0000", 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20193533899999999 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707424+0000", 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8697999999999999e-05 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.219 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:234 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.0981854860000002, 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.87684132699999995, 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 234 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:51:59.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9599999999999998e-07 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033821+0000", 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.653e-06 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033834+0000", 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3303e-05 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033847+0000", 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2327000000000001e-05 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808105+0000", 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.77425798199999996 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808280+0000", 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017508500000000001 2024-04-18T21:51:59.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.910658+0000", 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102378081 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:305 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086222+0000", 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.0457802430000001, 2024-04-18T21:51:59.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61042405399999999, 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 305 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086222+0000", 2024-04-18T21:51:59.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086222+0000", 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086225+0000", 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7499999999999999e-06 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086238+0000", 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3499000000000001e-05 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086273+0000", 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4610000000000002e-05 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086289+0000", 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6356000000000001e-05 2024-04-18T21:51:59.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086431+0000", 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014200000000000001 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086443+0000", 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1749999999999999e-05 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.190839+0000", 2024-04-18T21:51:59.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10439659 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.191013+0000", 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000173883 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.696646+0000", 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50563261599999998 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:306 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.045669212, 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71140329400000002, 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 306 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.230 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086334+0000", 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4900000000000004e-07 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086340+0000", 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0900000000000001e-06 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086359+0000", 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8940000000000002e-05 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086369+0000", 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7569999999999999e-06 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086608+0000", 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000239702 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086634+0000", 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5948999999999998e-05 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.291884+0000", 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20524995300000001 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292051+0000", 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000166797 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.797736+0000", 2024-04-18T21:51:59.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50568515700000005 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:307 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.0456191239999999, 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81219510800000005, 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.235 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 307 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086384+0000", 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9699999999999995e-07 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086388+0000", 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3900000000000003e-06 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086398+0000", 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8369999999999992e-06 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086406+0000", 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3599999999999996e-06 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086727+0000", 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032101900000000002 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086748+0000", 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0917e-05 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292674+0000", 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20592602600000001 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292826+0000", 2024-04-18T21:51:59.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015169700000000001 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.898578+0000", 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60575206500000001 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.240 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:59.242 INFO:tasks.rados.rados.0.smithi097.stdout:126: writing smithi09731216-35 from 1376256 to 1753088 tid 2 2024-04-18T21:51:59.243 INFO:tasks.rados.rados.0.smithi097.stdout:127: copy_from oid 31 from oid 2 current snap is 4 2024-04-18T21:51:59.243 INFO:tasks.rados.rados.0.smithi097.stdout:128: copy_from oid 21 from oid 20 current snap is 4 2024-04-18T21:51:59.243 INFO:tasks.rados.rados.0.smithi097.stdout:129: snap_create 2024-04-18T21:51:59.243 INFO:tasks.rados.rados.0.smithi097.stdout:124: finishing copy_from to smithi09731216-12 2024-04-18T21:51:59.243 INFO:tasks.rados.rados.0.smithi097.stdout:124: got expected ENOENT (src dne) 2024-04-18T21:51:59.243 INFO:tasks.rados.rados.0.smithi097.stdout:124: finishing copy_from racing read to smithi09731216-12 2024-04-18T21:51:59.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:51:59.244 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:51:59.239+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:51:59.342 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:59.339+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:59.375 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:59.375 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:59.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.375 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s5 42/31 ECSubRead(tid=97, to_read={3:c2fd059f:::smithi09731216-20:head=0,331776,0}, subchunks={3:c2fd059f:::smithi09731216-20:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:59.375 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:59.328564+0000", 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.047255919, 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.047304131999999999, 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "no flag points reached", 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.328564+0000", 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.328564+0000", 2024-04-18T21:51:59.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.328566+0000", 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.53e-06 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.328580+0000", 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3828e-05 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.328613+0000", 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3652000000000002e-05 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 1 2024-04-18T21:51:59.378 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:59.389 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:51:59.418 INFO:tasks.rados.rados.0.smithi097.stdout:99: finishing write tid 4 to smithi09731216-47 2024-04-18T21:51:59.419 INFO:tasks.rados.rados.0.smithi097.stdout:99: oid 47 updating version 0 to 11 2024-04-18T21:51:59.419 INFO:tasks.rados.rados.0.smithi097.stdout:99: oid 47 version 11 is already newer than 10 2024-04-18T21:51:59.419 INFO:tasks.rados.rados.0.smithi097.stdout:99: oid 47 updating version 11 to 12 2024-04-18T21:51:59.419 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 47 v 12 (ObjNum 58 snap 3 seq_num 58) dirty exists 2024-04-18T21:51:59.419 INFO:tasks.rados.rados.0.smithi097.stdout:99: left oid 47 (ObjNum 58 snap 3 seq_num 58) 2024-04-18T21:51:59.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:51:59.439+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:51:59.520 INFO:tasks.rados.rados.0.smithi097.stdout:102: finishing write tid 4 to smithi09731216-19 2024-04-18T21:51:59.520 INFO:tasks.rados.rados.0.smithi097.stdout:102: oid 19 updating version 0 to 16 2024-04-18T21:51:59.520 INFO:tasks.rados.rados.0.smithi097.stdout:102: oid 19 version 16 is already newer than 14 2024-04-18T21:51:59.520 INFO:tasks.rados.rados.0.smithi097.stdout:102: oid 19 version 16 is already newer than 15 2024-04-18T21:51:59.520 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 19 v 16 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-04-18T21:51:59.520 INFO:tasks.rados.rados.0.smithi097.stdout:102: left oid 19 (ObjNum 60 snap 3 seq_num 60) 2024-04-18T21:51:59.544 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:51:59.539+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:51:59.558 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:59.558 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:51:59.558 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:51:59.558 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:51:59.558 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:59.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:51:59.614 INFO:tasks.rados.rados.0.smithi097.stdout:126: finishing write tid 1 to smithi09731216-35 2024-04-18T21:51:59.623 INFO:tasks.rados.rados.0.smithi097.stdout:126: finishing write tid 2 to smithi09731216-35 2024-04-18T21:51:59.623 INFO:tasks.rados.rados.0.smithi097.stdout:126: finishing write tid 3 to smithi09731216-35 2024-04-18T21:51:59.664 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:51:59.663+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:51:59.715 INFO:tasks.rados.rados.0.smithi097.stdout:125: finishing write tid 1 to smithi09731216-38 2024-04-18T21:51:59.716 INFO:tasks.rados.rados.0.smithi097.stdout:125: finishing write tid 2 to smithi09731216-38 2024-04-18T21:51:59.729 INFO:tasks.rados.rados.0.smithi097.stdout:128: finishing copy_from racing read to smithi09731216-21 2024-04-18T21:51:59.743 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:51:59.743 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:51:59.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:51:59.743 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:51:59.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.744 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:100 3.cs0 3:348e44d8:::smithi09731216-34:head [append 0~557056,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:59.744 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:51:59.744 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.081858551, 2024-04-18T21:51:59.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50652407300000002, 2024-04-18T21:51:59.744 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.744 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.745 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.745 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.745 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.745 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.745 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100 2024-04-18T21:51:59.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659953+0000", 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.821e-06 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660381+0000", 2024-04-18T21:51:59.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042725800000000002 2024-04-18T21:51:59.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660395+0000", 2024-04-18T21:51:59.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.394e-05 2024-04-18T21:51:59.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.749 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660404+0000", 2024-04-18T21:51:59.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6520000000000001e-06 2024-04-18T21:51:59.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760564+0000", 2024-04-18T21:51:59.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10015995900000001 2024-04-18T21:51:59.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760699+0000", 2024-04-18T21:51:59.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013526 2024-04-18T21:51:59.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.752 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762105+0000", 2024-04-18T21:51:59.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00140595 2024-04-18T21:51:59.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965039+0000", 2024-04-18T21:51:59.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20293309400000001 2024-04-18T21:51:59.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166421+0000", 2024-04-18T21:51:59.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201382373 2024-04-18T21:51:59.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166476+0000", 2024-04-18T21:51:59.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4765999999999997e-05 2024-04-18T21:51:59.755 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.755 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.755 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.756 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:101 3.cs0 3:348e44d8:::smithi09731216-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:59.756 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:51:59.756 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.081388811, 2024-04-18T21:51:59.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60956795200000002, 2024-04-18T21:51:59.756 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.757 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.757 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.757 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.757 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.757 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.757 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101 2024-04-18T21:51:59.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.758 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:51:59.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:51:59.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660422+0000", 2024-04-18T21:51:59.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-07 2024-04-18T21:51:59.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660426+0000", 2024-04-18T21:51:59.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3830000000000002e-06 2024-04-18T21:51:59.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660436+0000", 2024-04-18T21:51:59.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8679999999999994e-06 2024-04-18T21:51:59.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660443+0000", 2024-04-18T21:51:59.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2999999999999998e-06 2024-04-18T21:51:59.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762872+0000", 2024-04-18T21:51:59.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1024294 2024-04-18T21:51:59.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762973+0000", 2024-04-18T21:51:59.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001015 2024-04-18T21:51:59.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763506+0000", 2024-04-18T21:51:59.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053209199999999996 2024-04-18T21:51:59.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965047+0000", 2024-04-18T21:51:59.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20154100999999999 2024-04-18T21:51:59.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269839+0000", 2024-04-18T21:51:59.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.304792377 2024-04-18T21:51:59.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269989+0000", 2024-04-18T21:51:59.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015033399999999999 2024-04-18T21:51:59.768 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.768 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.768 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.769 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:102 3.cs0 3:348e44d8:::smithi09731216-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:51:59.769 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:51:59.769 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.081339262, 2024-04-18T21:51:59.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.92759091000000005, 2024-04-18T21:51:59.769 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.770 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.770 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.770 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.770 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.770 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.770 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102 2024-04-18T21:51:59.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.771 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:51:59.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:51:59.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660472+0000", 2024-04-18T21:51:59.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:51:59.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660476+0000", 2024-04-18T21:51:59.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0879999999999997e-06 2024-04-18T21:51:59.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660493+0000", 2024-04-18T21:51:59.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6963e-05 2024-04-18T21:51:59.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660502+0000", 2024-04-18T21:51:59.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9579999999999996e-06 2024-04-18T21:51:59.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763793+0000", 2024-04-18T21:51:59.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10329121600000001 2024-04-18T21:51:59.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:51:59.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763891+0000", 2024-04-18T21:51:59.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8127000000000004e-05 2024-04-18T21:51:59.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372039+0000", 2024-04-18T21:51:59.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60814838199999999 2024-04-18T21:51:59.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372189+0000", 2024-04-18T21:51:59.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014991200000000001 2024-04-18T21:51:59.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.588062+0000", 2024-04-18T21:51:59.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.215872654 2024-04-18T21:51:59.779 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.5s0 30/26 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.981944774, 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30082227499999997, 2024-04-18T21:51:59.780 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759866+0000", 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.34e-07 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759873+0000", 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.973e-06 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759879+0000", 2024-04-18T21:51:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5440000000000004e-06 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759885+0000", 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3340000000000001e-06 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.060439+0000", 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30055447000000002 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.060688+0000", 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024862000000000002 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.4s3 30/26 ECSubWrite(tid=10, reqid=client.4509.0:97, at_version=30'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.760141+0000", 2024-04-18T21:51:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.981669224999999, 2024-04-18T21:51:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20499564100000001, 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760141+0000", 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760141+0000", 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760141+0000", 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.13e-07 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760254+0000", 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011241799999999999 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760263+0000", 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.217e-06 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760269+0000", 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5740000000000003e-06 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762841+0000", 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025722560000000002 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762852+0000", 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0821e-05 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965053+0000", 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20220110499999999 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965137+0000", 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3737000000000001e-05 2024-04-18T21:51:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.4s3 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.760281+0000", 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.981529463999999, 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20508437099999999, 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760281+0000", 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760281+0000", 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760281+0000", 2024-04-18T21:51:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9400000000000001e-07 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760291+0000", 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5840000000000001e-06 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760296+0000", 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2959999999999998e-06 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760306+0000", 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6339999999999996e-06 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.863516+0000", 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103210646 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.863542+0000", 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5812e-05 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965285+0000", 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10174351600000001 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965365+0000", 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9589e-05 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.976178233000001, 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299971873, 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1800000000000001e-07 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765635+0000", 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6450000000000001e-06 2024-04-18T21:51:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765644+0000", 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.755e-06 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.865734+0000", 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10008987699999999 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065540+0000", 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19980606200000001 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065604+0000", 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4115999999999994e-05 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.776822+0000", 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.964987971999999, 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.28932778799999997, 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776822+0000", 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776822+0000", 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776823+0000", 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8699999999999995e-07 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776826+0000", 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4860000000000002e-06 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.776836+0000", 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6549999999999999e-06 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.876933+0000", 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10009744 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.066102+0000", 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18916881299999999 2024-04-18T21:51:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.066150+0000", 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7806999999999998e-05 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.875115478, 2024-04-18T21:51:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299440857, 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:51:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.13e-07 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866703+0000", 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866722+0000", 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9522999999999999e-05 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966809+0000", 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10008695400000001 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166119+0000", 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19931021099999999 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166136+0000", 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6319000000000001e-05 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:78 3.15s0 3:ab33d1b5:::smithi09731216-26:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.760793838, 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1076682130000002, 2024-04-18T21:51:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981019+0000", 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.447e-06 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981025+0000", 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6810000000000001e-06 2024-04-18T21:51:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981044+0000", 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8856999999999999e-05 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981056+0000", 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1921e-05 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164029+0000", 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18297308600000001 2024-04-18T21:51:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164033+0000", 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0369999999999996e-06 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885490+0000", 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7214569900000001 2024-04-18T21:51:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885771+0000", 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028064300000000002 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.088685+0000", 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.202913551 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 31/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.065688+0000", 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.676122253999999, 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20463684800000001, 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065688+0000", 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065688+0000", 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065690+0000", 2024-04-18T21:51:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8729999999999999e-06 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065701+0000", 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1677e-05 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065728+0000", 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.669e-05 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065752+0000", 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4196e-05 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269705+0000", 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20395275299999999 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.270325+0000", 2024-04-18T21:51:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00061965900000000005 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.cs0 31/26 ECSubRead(tid=23, to_read={3:348e44d8:::smithi09731216-34:head=0,4096,0}, subchunks={3:348e44d8:::smithi09731216-34:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.369168396999999, 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214868642, 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372642+0000", 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372695+0000", 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3584000000000001e-05 2024-04-18T21:51:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.472813+0000", 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10011788200000001 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.587035+0000", 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.114221857 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.587510+0000", 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047531899999999998 2024-04-18T21:51:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.13s5 32/31 ECSubWrite(tid=14, reqid=client.4509.0:73, at_version=32'10, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:48.593977+0000", 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.147833629000001, 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.204113025, 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593977+0000", 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593977+0000", 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593979+0000", 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1569999999999998e-06 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.594255+0000", 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027672299999999999 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.594300+0000", 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5086000000000002e-05 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.694435+0000", 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100134078 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.794654+0000", 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100219603 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.794723+0000", 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8978000000000003e-05 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798007+0000", 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032838590000000001 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798090+0000", 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2540999999999995e-05 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:133 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.731527841, 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.393294378, 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 133 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010286+0000", 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012871+0000", 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025854699999999999 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012896+0000", 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5409e-05 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012911+0000", 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.484e-05 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013086+0000", 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017422499999999999 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013437+0000", 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035166100000000002 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.016765+0000", 2024-04-18T21:51:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033273949999999999 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.020122+0000", 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033576370000000001 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403436+0000", 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38331330800000002 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403577+0000", 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014114999999999999 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:134 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.728826314999999, 2024-04-18T21:51:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39119692299999997, 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 134 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:51:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012986+0000", 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0839999999999999e-06 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012995+0000", 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4159999999999993e-06 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013025+0000", 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0035999999999998e-05 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013046+0000", 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0720000000000002e-05 2024-04-18T21:51:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.017993+0000", 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049471020000000001 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.018258+0000", 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026496000000000001 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.019560+0000", 2024-04-18T21:51:59.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001301373 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.020585+0000", 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010257910000000001 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403896+0000", 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38331068899999998 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.404181+0000", 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028475199999999999 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:235 3.cs0 3:3bc99ad3:::smithi09731216-27:head [read 0~442368,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033116+0000", 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.7086945589999996, 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.204192295, 2024-04-18T21:51:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033116+0000", 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033116+0000", 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033117+0000", 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.717e-06 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033125+0000", 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9319999999999996e-06 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033146+0000", 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0424e-05 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033154+0000", 2024-04-18T21:51:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7870000000000007e-06 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033222+0000", 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7574e-05 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033230+0000", 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.405e-06 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.035364+0000", 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00213393 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.035439+0000", 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4649999999999998e-05 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.237308+0000", 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201868876 2024-04-18T21:51:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:236 3.cs0 3:3bc99ad3:::smithi09731216-27:head [read 0~442368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033177+0000", 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.7086331560000003, 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20630516600000001, 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 236 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033177+0000", 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033177+0000", 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033177+0000", 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9500000000000003e-07 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033181+0000", 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9260000000000002e-06 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033189+0000", 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1489999999999997e-06 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033194+0000", 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7599999999999999e-06 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033373+0000", 2024-04-18T21:51:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017826300000000001 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033393+0000", 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0460000000000001e-05 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.135792+0000", 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102399357 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.135881+0000", 2024-04-18T21:51:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8745000000000005e-05 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.239482+0000", 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10360101100000001 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:237 3.cs0 3:3bc99ad3:::smithi09731216-27:head [read 0~442368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.7085657449999996, 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30619164900000001, 2024-04-18T21:51:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033245+0000", 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3499999999999996e-07 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033248+0000", 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3730000000000001e-06 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033256+0000", 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8509999999999999e-06 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033262+0000", 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4480000000000002e-06 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033435+0000", 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000173847 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033439+0000", 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6540000000000001e-06 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.237425+0000", 2024-04-18T21:51:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20398589 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.237506+0000", 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1211000000000006e-05 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.339436+0000", 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10192983999999999 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 40/26 ECSubWriteReply(tid=52, last_complete=40'20, committed=1, applied=1))", 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:57.521559+0000", 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 2.220250821, 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20765846399999999, 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521559+0000", 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521559+0000", 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521561+0000", 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.576e-06 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521570+0000", 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.687e-06 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521593+0000", 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3791999999999999e-05 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521606+0000", 2024-04-18T21:51:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.225e-05 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.728942+0000", 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.207335928 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.729218+0000", 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027623099999999999 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 40/26 ECSubWriteReply(tid=53, last_complete=40'21, committed=1, applied=1))", 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 2.2201852789999998, 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.409140051, 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0900000000000002e-07 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521629+0000", 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0169999999999998e-06 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521640+0000", 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0246e-05 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:51:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521647+0000", 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9070000000000001e-06 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.929171+0000", 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40752403300000001 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.930765+0000", 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001594339 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:51:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:51:59.862 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:51:59.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:51:59.864 INFO:tasks.rados.rados.0.smithi097.stdout:125: finishing write tid 3 to smithi09731216-38 2024-04-18T21:51:59.864 INFO:tasks.rados.rados.0.smithi097.stdout:127: finishing copy_from to smithi09731216-31 2024-04-18T21:51:59.864 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 31 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-18T21:51:59.864 INFO:tasks.rados.rados.0.smithi097.stdout:127: finishing copy_from racing read to smithi09731216-31 2024-04-18T21:51:59.864 INFO:tasks.rados.rados.0.smithi097.stdout:126: finishing write tid 4 to smithi09731216-35 2024-04-18T21:51:59.864 INFO:tasks.rados.rados.0.smithi097.stdout:126: oid 35 updating version 0 to 16 2024-04-18T21:51:59.864 INFO:tasks.rados.rados.0.smithi097.stdout:126: oid 35 version 16 is already newer than 14 2024-04-18T21:51:59.865 INFO:tasks.rados.rados.0.smithi097.stdout:126: oid 35 version 16 is already newer than 15 2024-04-18T21:51:59.865 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 35 v 16 (ObjNum 70 snap 4 seq_num 70) dirty exists 2024-04-18T21:51:59.865 INFO:tasks.rados.rados.0.smithi097.stdout:126: left oid 35 (ObjNum 70 snap 4 seq_num 70) 2024-04-18T21:51:59.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:51:59.866 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:51:59.859+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:51:59.932 INFO:tasks.rados.rados.0.smithi097.stdout:128: finishing copy_from to smithi09731216-21 2024-04-18T21:51:59.933 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 21 v 39 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-04-18T21:51:59.964 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:51:59.959+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:00.019 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:00.019 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:00.020 INFO:tasks.rados.rados.0.smithi097.stdout:125: finishing write tid 4 to smithi09731216-38 2024-04-18T21:52:00.020 INFO:tasks.rados.rados.0.smithi097.stdout:125: oid 38 updating version 0 to 52 2024-04-18T21:52:00.020 INFO:tasks.rados.rados.0.smithi097.stdout:125: oid 38 version 52 is already newer than 50 2024-04-18T21:52:00.020 INFO:tasks.rados.rados.0.smithi097.stdout:125: oid 38 version 52 is already newer than 51 2024-04-18T21:52:00.020 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 38 v 52 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-04-18T21:52:00.021 INFO:tasks.rados.rados.0.smithi097.stdout:125: left oid 38 (ObjNum 69 snap 4 seq_num 69) 2024-04-18T21:52:00.035 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:00.036 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:00.039 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:00.039 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:00.040 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:00.040 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:00.052 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:00.052 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:00.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:00.057 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.4 to 1 (10000) 2024-04-18T21:52:00.060 INFO:tasks.rados.rados.0.smithi097.stdout:99: done (7 left) 2024-04-18T21:52:00.060 INFO:tasks.rados.rados.0.smithi097.stdout:102: done (6 left) 2024-04-18T21:52:00.060 INFO:tasks.rados.rados.0.smithi097.stdout:124: done (5 left) 2024-04-18T21:52:00.060 INFO:tasks.rados.rados.0.smithi097.stdout:125: done (4 left) 2024-04-18T21:52:00.060 INFO:tasks.rados.rados.0.smithi097.stdout:126: done (3 left) 2024-04-18T21:52:00.061 INFO:tasks.rados.rados.0.smithi097.stdout:127: done (2 left) 2024-04-18T21:52:00.061 INFO:tasks.rados.rados.0.smithi097.stdout:128: done (1 left) 2024-04-18T21:52:00.061 INFO:tasks.rados.rados.0.smithi097.stdout:129: done (0 left) 2024-04-18T21:52:00.061 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 9 current snap is 5 2024-04-18T21:52:00.061 INFO:tasks.rados.rados.0.smithi097.stdout:130: seq_num 71 ranges {1245184=425984,1671168=573440,2244608=196608} 2024-04-18T21:52:00.070 INFO:tasks.rados.rados.0.smithi097.stdout:130: writing smithi09731216-9 from 1245184 to 1671168 tid 1 2024-04-18T21:52:00.073 INFO:tasks.rados.rados.0.smithi097.stdout:130: writing smithi09731216-9 from 1671168 to 2244608 tid 2 2024-04-18T21:52:00.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-18T21:52:00.075 INFO:tasks.rados.rados.0.smithi097.stdout:130: writing smithi09731216-9 from 2244608 to 2441216 tid 3 2024-04-18T21:52:00.075 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 12 current snap is 5 2024-04-18T21:52:00.075 INFO:tasks.rados.rados.0.smithi097.stdout:131: seq_num 72 ranges {196608=442368,638976=720896,1359872=49152} 2024-04-18T21:52:00.078 INFO:tasks.rados.rados.0.smithi097.stdout:131: writing smithi09731216-12 from 196608 to 638976 tid 1 2024-04-18T21:52:00.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-18T21:52:00.083 INFO:tasks.rados.rados.0.smithi097.stdout:131: writing smithi09731216-12 from 638976 to 1359872 tid 2 2024-04-18T21:52:00.083 INFO:tasks.rados.rados.0.smithi097.stdout:131: writing smithi09731216-12 from 1359872 to 1409024 tid 3 2024-04-18T21:52:00.083 INFO:tasks.rados.rados.0.smithi097.stdout:132: rollback oid 22 current snap is 5 2024-04-18T21:52:00.083 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 22 to 0 2024-04-18T21:52:00.083 INFO:tasks.rados.rados.0.smithi097.stdout:133: snap_create 2024-04-18T21:52:00.096 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:00.095+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:00.166 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:00.163+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:00.177 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:00.177 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:00.178 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:00.178 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:00.178 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:00.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:00.266 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:00.263+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:00.367 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:00.367 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:00.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:00.367 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:00.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.11s6 32/31 ECSubWrite(tid=9, reqid=client.4509.0:127, at_version=32'5, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:48.899509+0000", 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.465534912000001, 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20351012600000001, 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899509+0000", 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899509+0000", 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899510+0000", 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.807e-06 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899962+0000", 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045185800000000002 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.899992+0000", 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9949000000000001e-05 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.000095+0000", 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10010311299999999 2024-04-18T21:52:00.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.100305+0000", 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10020960800000001 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.100322+0000", 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6657e-05 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.102893+0000", 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025716609999999998 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.103019+0000", 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000125473 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 36/26 ECSubWrite(tid=36, reqid=client.4509.0:216, at_version=36'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.3464982010000002, 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30205913200000001, 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:00.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.35e-07 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018570+0000", 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.438e-05 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018583+0000", 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2928999999999999e-05 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118702+0000", 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1001194 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218952+0000", 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10025022 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218966+0000", 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3709e-05 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:00.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320525+0000", 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10155911099999999 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320604+0000", 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9147999999999995e-05 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 36/26 ECSubWrite(tid=37, reqid=client.4509.0:217, at_version=36'13, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.118770+0000", 2024-04-18T21:52:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.2462735980000001, 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20354913499999999, 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118770+0000", 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118770+0000", 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118771+0000", 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.345e-06 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118799+0000", 2024-04-18T21:52:00.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7997999999999999e-05 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118812+0000", 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3286e-05 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218952+0000", 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100139298 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219853+0000", 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00090166699999999998 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.219867+0000", 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3304e-05 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.322235+0000", 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10236814900000001 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.322319+0000", 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4087999999999999e-05 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:239 3.as0 3:5bb4af71:::smithi09731216-7:head [delete] snapc 4=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.3510112699999999, 2024-04-18T21:52:00.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40857630099999998, 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014034+0000", 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2500000000000001e-06 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014044+0000", 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1889999999999996e-06 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014067+0000", 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3251000000000002e-05 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114145+0000", 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078193 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114306+0000", 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161044 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114320+0000", 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4044e-05 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317101+0000", 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20278075300000001 2024-04-18T21:52:00.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317362+0000", 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026128799999999998 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.319467+0000", 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021048719999999998 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.326280+0000", 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006813375 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422400+0000", 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096119880000000005 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422608+0000", 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020816199999999999 2024-04-18T21:52:00.388 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:308 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~737280] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 2.274571194, 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69365002799999997, 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:00.389 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 308 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090475+0000", 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.976e-06 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091768+0000", 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00129317 2024-04-18T21:52:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091803+0000", 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4502999999999998e-05 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091826+0000", 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3235999999999999e-05 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270501+0000", 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.178674888 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270680+0000", 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017917599999999999 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.273556+0000", 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028757610000000001 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281580+0000", 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080240799999999994 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783886+0000", 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50230573000000001 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784122+0000", 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023650799999999999 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:309 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 2.2724902010000001, 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69216249600000002, 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.395 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 309 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092555+0000", 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093122+0000", 2024-04-18T21:52:00.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056744499999999997 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093146+0000", 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128999999999999e-05 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093165+0000", 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8300000000000001e-05 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275352+0000", 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18218714899999999 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275632+0000", 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028059100000000001 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.277549+0000", 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001917087 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281584+0000", 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004034513 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784502+0000", 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50291762600000001 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784716+0000", 2024-04-18T21:52:00.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021415599999999999 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:310 3.as0 3:5bb4af71:::smithi09731216-7:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 2.2718490309999999, 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69187687499999995, 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.401 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 310 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093196+0000", 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093210+0000", 2024-04-18T21:52:00.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3517e-05 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093227+0000", 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7050000000000001e-05 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093243+0000", 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5965999999999999e-05 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279008+0000", 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.185765398 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279287+0000", 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027941400000000002 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.280700+0000", 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412581 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482172+0000", 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20147153900000001 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784956+0000", 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30278430299999998 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785071+0000", 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115454 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:311 3.as0 3:5bb4af71:::smithi09731216-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 2.2717740790000001, 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99553992999999996, 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.407 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 311 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093271+0000", 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093278+0000", 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1459999999999999e-06 2024-04-18T21:52:00.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093295+0000", 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7408999999999999e-05 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093328+0000", 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2276000000000001e-05 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785901+0000", 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.692573628 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.786181+0000", 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027934999999999999 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.088809+0000", 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30262875700000003 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=14, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.370158+0000", 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.994885676, 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21261686199999999, 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.412 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370158+0000", 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370158+0000", 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370159+0000", 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3599999999999999e-06 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370167+0000", 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0430000000000008e-06 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370188+0000", 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0599999999999999e-05 2024-04-18T21:52:00.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370202+0000", 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4392999999999999e-05 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582740+0000", 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.212537737 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582775+0000", 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4728999999999999e-05 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.370223+0000", 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.9948206339999999, 2024-04-18T21:52:00.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21269959699999999, 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370223+0000", 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370223+0000", 2024-04-18T21:52:00.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370223+0000", 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7000000000000005e-07 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370227+0000", 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.433e-06 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370237+0000", 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370244+0000", 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5399999999999998e-06 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582886+0000", 2024-04-18T21:52:00.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.212642205 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582922+0000", 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6099000000000001e-05 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.375536+0000", 2024-04-18T21:52:00.420 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.989507514, 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20745433599999999, 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.375536+0000", 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.375536+0000", 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.375537+0000", 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.353e-06 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.375546+0000", 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5129999999999996e-06 2024-04-18T21:52:00.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.375569+0000", 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2877000000000001e-05 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.375582+0000", 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3628999999999999e-05 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582968+0000", 2024-04-18T21:52:00.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20738549100000001 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582990+0000", 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2473e-05 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=13, last_complete=41'9, committed=1, applied=1))", 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.377897+0000", 2024-04-18T21:52:00.424 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.9871465289999999, 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.205150999, 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.377897+0000", 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.377897+0000", 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.377899+0000", 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8929999999999999e-06 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.377910+0000", 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1333e-05 2024-04-18T21:52:00.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.377940+0000", 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9929000000000001e-05 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.377954+0000", 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.434e-05 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.583026+0000", 2024-04-18T21:52:00.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.205071735 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.583048+0000", 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1769000000000001e-05 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=13, last_complete=41'9, committed=1, applied=1))", 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.378527+0000", 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.986515939, 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20457924499999999, 2024-04-18T21:52:00.428 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.378527+0000", 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.378527+0000", 2024-04-18T21:52:00.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.378529+0000", 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.66e-06 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.378540+0000", 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1178999999999999e-05 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.378564+0000", 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.351e-05 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.378588+0000", 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4185999999999998e-05 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.583082+0000", 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20449440399999999 2024-04-18T21:52:00.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.583107+0000", 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4306000000000001e-05 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.9849546249999999, 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30330003799999999, 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.432 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380090+0000", 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.47e-07 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380096+0000", 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6100000000000002e-06 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380115+0000", 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8986000000000001e-05 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380129+0000", 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4229e-05 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683350+0000", 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30322019500000003 2024-04-18T21:52:00.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683389+0000", 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9171000000000002e-05 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=14, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.9837120910000001, 2024-04-18T21:52:00.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30209698499999998, 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:00.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381333+0000", 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.843e-06 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381343+0000", 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8700000000000004e-06 2024-04-18T21:52:00.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381368+0000", 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4613e-05 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381381+0000", 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2857000000000001e-05 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683415+0000", 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30203418999999998 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683428+0000", 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3611999999999999e-05 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.440 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.382104+0000", 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.9829394730000001, 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.301352913, 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382104+0000", 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382104+0000", 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382105+0000", 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.443e-06 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382114+0000", 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2130000000000008e-06 2024-04-18T21:52:00.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382133+0000", 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9830000000000002e-05 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482231+0000", 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10009760600000001 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683445+0000", 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20121439399999999 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683457+0000", 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1426999999999999e-05 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=13, last_complete=41'9, committed=1, applied=1))", 2024-04-18T21:52:00.444 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.8824791809999999, 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30158400099999999, 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482566+0000", 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4360000000000001e-06 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482574+0000", 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4700000000000002e-06 2024-04-18T21:52:00.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482593+0000", 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9015999999999999e-05 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482608+0000", 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5023e-05 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783615+0000", 2024-04-18T21:52:00.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30100726999999999 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784148+0000", 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053278600000000005 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=14, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:00.448 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.88239081, 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30208896099999999, 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5000000000000002e-07 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482657+0000", 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1269999999999996e-06 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482672+0000", 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4688999999999999e-05 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482683+0000", 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1165999999999999e-05 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784263+0000", 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30157952599999999 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784742+0000", 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047880299999999998 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.8823464780000001, 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30306911800000003, 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:00.452 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:00.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482701+0000", 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1470000000000001e-06 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482712+0000", 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.182e-05 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482719+0000", 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9129999999999997e-06 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784839+0000", 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30211942400000003 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785766+0000", 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092735400000000003 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.4.19:4 3.as0 3:561bcae7:::smithi09731216-2:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:59.327830+0000", 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.03721336, 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.24013599099999999, 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327830+0000", 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327830+0000", 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327832+0000", 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0480000000000001e-06 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327839+0000", 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9600000000000003e-06 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327851+0000", 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2317000000000001e-05 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327861+0000", 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1689999999999998e-06 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327941+0000", 2024-04-18T21:52:00.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9963000000000007e-05 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.328016+0000", 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5492000000000002e-05 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.567966+0000", 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.239950042 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:00.460 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:00.460 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:00.460 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:00.461 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:00.366+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:00.461 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:00.461 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:00.462 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:00.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:00.463 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:00.463 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:00.463 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:00.463 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:00.464 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:00.464 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:00.467 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:00.463+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:00.481 INFO:tasks.rados.rados.0.smithi097.stdout:132: finishing rollback tid 0 to smithi09731216-22 2024-04-18T21:52:00.481 INFO:tasks.rados.rados.0.smithi097.stdout:132: finishing rollback tid 1 to smithi09731216-22 2024-04-18T21:52:00.481 INFO:tasks.rados.rados.0.smithi097.stdout:132: finishing rollback tid 2 to smithi09731216-22 2024-04-18T21:52:00.481 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 22 v 56 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-18T21:52:00.483 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:00.483 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:00.483 INFO:tasks.rados.rados.0.smithi097.stdout:130: finishing write tid 1 to smithi09731216-9 2024-04-18T21:52:00.484 INFO:tasks.rados.rados.0.smithi097.stdout:130: finishing write tid 2 to smithi09731216-9 2024-04-18T21:52:00.485 INFO:tasks.rados.rados.0.smithi097.stdout:130: finishing write tid 3 to smithi09731216-9 2024-04-18T21:52:00.505 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:00.505 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:00.526 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:00.526 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:00.547 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:00.547 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:00.569 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:00.566+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:00.590 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:00.590 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:00.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:00.652 INFO:tasks.rados.rados.0.smithi097.stdout:131: finishing write tid 1 to smithi09731216-12 2024-04-18T21:52:00.652 INFO:tasks.rados.rados.0.smithi097.stdout:131: finishing write tid 2 to smithi09731216-12 2024-04-18T21:52:00.652 INFO:tasks.rados.rados.0.smithi097.stdout:131: finishing write tid 3 to smithi09731216-12 2024-04-18T21:52:00.652 INFO:tasks.rados.rados.0.smithi097.stdout:131: finishing write tid 4 to smithi09731216-12 2024-04-18T21:52:00.670 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:00.667+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:00.688 INFO:tasks.rados.rados.0.smithi097.stdout:130: finishing write tid 4 to smithi09731216-9 2024-04-18T21:52:00.729 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:00.729 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:00.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:00.784 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:00.783+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:00.791 INFO:tasks.rados.rados.0.smithi097.stdout:130: finishing write tid 5 to smithi09731216-9 2024-04-18T21:52:00.791 INFO:tasks.rados.rados.0.smithi097.stdout:130: oid 9 updating version 0 to 28 2024-04-18T21:52:00.791 INFO:tasks.rados.rados.0.smithi097.stdout:130: oid 9 version 28 is already newer than 27 2024-04-18T21:52:00.791 INFO:tasks.rados.rados.0.smithi097.stdout:130: oid 9 updating version 28 to 30 2024-04-18T21:52:00.791 INFO:tasks.rados.rados.0.smithi097.stdout:130: oid 9 version 30 is already newer than 29 2024-04-18T21:52:00.791 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 9 v 30 (ObjNum 71 snap 5 seq_num 71) dirty exists 2024-04-18T21:52:00.792 INFO:tasks.rados.rados.0.smithi097.stdout:130: left oid 9 (ObjNum 71 snap 5 seq_num 71) 2024-04-18T21:52:00.871 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:00.867+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:00.885 INFO:tasks.rados.rados.0.smithi097.stdout:131: finishing write tid 5 to smithi09731216-12 2024-04-18T21:52:00.885 INFO:tasks.rados.rados.0.smithi097.stdout:131: oid 12 updating version 0 to 21 2024-04-18T21:52:00.885 INFO:tasks.rados.rados.0.smithi097.stdout:131: oid 12 updating version 21 to 24 2024-04-18T21:52:00.886 INFO:tasks.rados.rados.0.smithi097.stdout:131: oid 12 version 24 is already newer than 23 2024-04-18T21:52:00.886 INFO:tasks.rados.rados.0.smithi097.stdout:131: oid 12 version 24 is already newer than 22 2024-04-18T21:52:00.886 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 12 v 24 (ObjNum 72 snap 5 seq_num 72) dirty exists 2024-04-18T21:52:00.886 INFO:tasks.rados.rados.0.smithi097.stdout:131: left oid 12 (ObjNum 72 snap 5 seq_num 72) 2024-04-18T21:52:00.901 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:00.902 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:00.917 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:00.972 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:00.967+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:01.049 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:01.049 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:01.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:01.072 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is unset 2024-04-18T21:52:01.072 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.5 to 1 (10000) 2024-04-18T21:52:01.078 INFO:tasks.rados.rados.0.smithi097.stdout:130: done (3 left) 2024-04-18T21:52:01.078 INFO:tasks.rados.rados.0.smithi097.stdout:131: done (2 left) 2024-04-18T21:52:01.078 INFO:tasks.rados.rados.0.smithi097.stdout:132: done (1 left) 2024-04-18T21:52:01.078 INFO:tasks.rados.rados.0.smithi097.stdout:133: done (0 left) 2024-04-18T21:52:01.078 INFO:tasks.rados.rados.0.smithi097.stdout:134: read oid 33 snap -1 2024-04-18T21:52:01.078 INFO:tasks.rados.rados.0.smithi097.stdout:134: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-18T21:52:01.078 INFO:tasks.rados.rados.0.smithi097.stdout:135: read oid 45 snap -1 2024-04-18T21:52:01.079 INFO:tasks.rados.rados.0.smithi097.stdout:135: expect deleted 2024-04-18T21:52:01.079 INFO:tasks.rados.rados.0.smithi097.stdout:136: snap_create 2024-04-18T21:52:01.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-18T21:52:01.107 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:01.106+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:01.174 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:01.171+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:01.187 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:01.188 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:01.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:01.274 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:01.271+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:01.368 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:01.368 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:01.375 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:01.371+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:01.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:01.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:01.475+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:01.504 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:01.504 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:01.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:01.564 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:01.559+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:01.578 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:01.575+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:01.584 INFO:tasks.rados.rados.0.smithi097.stdout:134: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-18T21:52:01.636 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:01.637 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:01.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:01.695 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:01.694+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:01.768 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:01.768 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:01.780 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:01.775+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:01.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:01.880 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:01.878+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:01.951 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:01.951 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:01.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:01.981 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:01.979+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:02.076 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.6 to 1 (10000) 2024-04-18T21:52:02.082 INFO:tasks.rados.rados.0.smithi097.stdout:134: done (2 left) 2024-04-18T21:52:02.082 INFO:tasks.rados.rados.0.smithi097.stdout:135: done (1 left) 2024-04-18T21:52:02.082 INFO:tasks.rados.rados.0.smithi097.stdout:136: done (0 left) 2024-04-18T21:52:02.082 INFO:tasks.rados.rados.0.smithi097.stdout:137: copy_from oid 48 from oid 42 current snap is 7 2024-04-18T21:52:02.082 INFO:tasks.rados.rados.0.smithi097.stdout:138: delete oid 29 current snap is 7 2024-04-18T21:52:02.082 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:02.079+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:02.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-18T21:52:02.130 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:02.131 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:02.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:02.270 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:02.270 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:02.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:02.288 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:02.191+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:02.288 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:02.283+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:02.384 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.379+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.385 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.379+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:02.406 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:02.406 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:02.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:02.486 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:02.482+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:02.508 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.509 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.483+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.510 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.511 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.512 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.514 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.514 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.514 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.514 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.514 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.514 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.514 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.514 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.515 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.487+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.516 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.517 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.518 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.518 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.518 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.518 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.518 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.518 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.519 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.520 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.521 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.522 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.523 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.491+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.524 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.525 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.526 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.495+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.539 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:02.540 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:02.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:02.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:02.585 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.585 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.586 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.587 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.587 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.587 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.579+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.587 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.587 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.587 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.587 INFO:tasks.rados.rados.0.smithi097.stdout:138: done (1 left) 2024-04-18T21:52:02.587 INFO:tasks.rados.rados.0.smithi097.stdout:139: snap_create 2024-04-18T21:52:02.589 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.590 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.591 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.592 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.593 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.594 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.594 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.594 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.594 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.583+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.595 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.595 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.595 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.596 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.591+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.597 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:02.591+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.600 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.601 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.602 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.603 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.603 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.603 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.603 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.603 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.595+0000 7f8625ea1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:02.688 INFO:tasks.rados.rados.0.smithi097.stdout:137: finishing copy_from to smithi09731216-48 2024-04-18T21:52:02.688 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 48 v 13 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-18T21:52:02.690 INFO:tasks.rados.rados.0.smithi097.stdout:137: finishing copy_from racing read to smithi09731216-48 2024-04-18T21:52:02.696 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:02.694+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:02.716 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:02.716 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:02.717 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:02.718 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:02.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:02.734 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:02.735 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:02.751 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:02.752 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:02.769 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:02.769 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:02.797 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:02.795+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:02.847 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:02.847 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:02.848 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:02.848 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:02.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:02.890 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:02.890 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:02.898 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:02.895+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:02.911 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:02.911 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:02.933 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:02.933 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:02.959 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:02.959 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:02.985 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:02.986 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:02.999 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:02.995+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:03.011 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:03.011 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:03.021 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:03.021 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:03.021 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:03.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:03.039 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:03.040 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:03.062 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:03.062 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:03.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-18T21:52:03.100 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:03.098+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:03.157 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:03.157 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:03.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:03.157 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:03.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:155 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'109, mlcod=24'105)", 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.850432215000001, 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30724263200000002, 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6800000000000002e-07 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306525+0000", 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2228999999999999e-05 2024-04-18T21:52:03.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306532+0000", 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7469999999999999e-06 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306539+0000", 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1250000000000004e-06 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.307270+0000", 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00073067900000000003 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.307309+0000", 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9830000000000003e-05 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613662+0000", 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30635288799999999 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613735+0000", 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2866000000000003e-05 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:156 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'110, mlcod=24'105)", 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.850379442000001, 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30725738200000002, 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.162 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306546+0000", 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.16e-07 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306548+0000", 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.672e-06 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306553+0000", 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0730000000000004e-06 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306556+0000", 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.959e-06 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408193+0000", 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101636647 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408269+0000", 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5767999999999997e-05 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613742+0000", 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.205473081 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613803+0000", 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0965999999999999e-05 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.166 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:157 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'111, mlcod=24'105)", 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.850364479, 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30730748800000002, 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.72e-07 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306563+0000", 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2730000000000001e-06 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306572+0000", 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2390000000000008e-06 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306577+0000", 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7709999999999997e-06 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408635+0000", 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102058725 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408654+0000", 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8921000000000001e-05 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613808+0000", 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20515377100000001 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613868+0000", 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9616e-05 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7944477320000001, 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40570043900000002, 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.171 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362478+0000", 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4279999999999999e-06 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362656+0000", 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017710100000000001 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362673+0000", 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7802e-05 2024-04-18T21:52:03.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462745+0000", 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100071484 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.767971+0000", 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30522610500000003 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.768177+0000", 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020651900000000001 2024-04-18T21:52:03.174 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7275584960000003, 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44082012100000001, 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:03.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:03.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429367+0000", 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5300000000000004e-07 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429643+0000", 2024-04-18T21:52:03.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027611500000000001 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429665+0000", 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1603e-05 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.529793+0000", 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100128809 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868616+0000", 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.33882269300000001 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870186+0000", 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015703480000000001 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:03.180 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.694088625, 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40574496100000002, 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0100000000000001e-07 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462933+0000", 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6247000000000002e-05 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462943+0000", 2024-04-18T21:52:03.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.944e-06 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462948+0000", 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4979999999999997e-06 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868491+0000", 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40554244699999997 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868581+0000", 2024-04-18T21:52:03.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0524000000000003e-05 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6911782200000003, 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71932642099999999, 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9599999999999998e-07 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465774+0000", 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6846e-05 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465781+0000", 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2640000000000002e-06 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465785+0000", 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7759999999999999e-06 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.185006+0000", 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71922088500000003 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.185073+0000", 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6853999999999994e-05 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:03.187 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6911092080000003, 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82169303100000002, 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:03.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4000000000000002e-07 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465848+0000", 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1609000000000002e-05 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465853+0000", 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.366e-06 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465856+0000", 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.151e-06 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286522+0000", 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82066616299999995 2024-04-18T21:52:03.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.287509+0000", 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0009863020000000001 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6910124030000002, 2024-04-18T21:52:03.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40453295099999997, 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465913+0000", 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6199999999999998e-07 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465996+0000", 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3168000000000001e-05 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.466007+0000", 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0662000000000001e-05 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.466014+0000", 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8639999999999998e-06 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870275+0000", 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.404261702 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870445+0000", 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016999300000000001 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.195 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6255443039999999, 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.75509254699999995, 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531387+0000", 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2899999999999999e-06 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531557+0000", 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016995899999999999 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531588+0000", 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1069999999999999e-05 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531606+0000", 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8479999999999999e-05 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286385+0000", 2024-04-18T21:52:03.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.754778964 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286473+0000", 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7783999999999998e-05 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:03.199 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.5908346770000001, 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858464, 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:03.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566092+0000", 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5799999999999999e-06 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566247+0000", 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155613 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566277+0000", 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0111999999999999e-05 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566315+0000", 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7141999999999998e-05 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.974376+0000", 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.408061795 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.974675+0000", 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029839800000000001 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=7, attrs_read=0))", 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.1501577440000004, 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41262958999999999, 2024-04-18T21:52:03.203 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006768+0000", 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4139999999999999e-06 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006798+0000", 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9499e-05 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006823+0000", 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4706999999999999e-05 2024-04-18T21:52:03.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006842+0000", 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9507000000000001e-05 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418445+0000", 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41160240300000001 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419397+0000", 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095206000000000002 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.10s1 41/39 ECSubRead(tid=11, to_read={3:0e754656:::smithi09731216-19:head=0,4096,0}, subchunks={3:0e754656:::smithi09731216-19:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.1353341400000003, 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39881851499999998, 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.207 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:03.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021616+0000", 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5575000000000002e-05 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021627+0000", 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1099e-05 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.420045+0000", 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39841796699999998 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.420409+0000", 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036387399999999998 2024-04-18T21:52:03.210 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.1346831509999999, 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39759610699999998, 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6199999999999997e-07 2024-04-18T21:52:03.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022253+0000", 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0283e-05 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022265+0000", 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2302e-05 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022271+0000", 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2550000000000003e-06 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419761+0000", 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39748950500000002 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419838+0000", 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7000000000000001e-05 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.1344593789999999, 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39718676600000002, 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.215 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022467+0000", 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3030000000000001e-06 2024-04-18T21:52:03.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022485+0000", 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8363999999999999e-05 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022504+0000", 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9283000000000002e-05 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022514+0000", 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8430000000000005e-06 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419513+0000", 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39699880100000001 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419652+0000", 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013917199999999999 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.218 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.1343405029999998, 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39738200800000001, 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:03.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022585+0000", 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7599999999999996e-07 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022601+0000", 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6203000000000001e-05 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022613+0000", 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1334e-05 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022618+0000", 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4110000000000004e-06 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419896+0000", 2024-04-18T21:52:03.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39727772700000002 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419966+0000", 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0556999999999995e-05 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.10s1 41/39 ECSubWriteReply(tid=10, last_complete=41'16, committed=1, applied=1))", 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:03.222 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.134287907, 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39534124999999998, 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022640+0000", 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0029999999999999e-06 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022652+0000", 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1585e-05 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022656+0000", 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5739999999999997e-06 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.417932+0000", 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39527527800000001 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.417978+0000", 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6350000000000002e-05 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.10s1 41/39 ECSubWriteReply(tid=10, last_complete=41'16, committed=1, applied=1))", 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.052227609, 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31366916900000003, 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.226 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104699+0000", 2024-04-18T21:52:03.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.686e-06 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104713+0000", 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4296e-05 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104742+0000", 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9204999999999999e-05 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104758+0000", 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5254e-05 2024-04-18T21:52:03.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418210+0000", 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31345283600000001 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418366+0000", 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155892 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.229 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 3.18s6,from:0'0,to:38'13,epoch:41/31,start:3:18000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.0507990850000004, 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31261068600000003, 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106127+0000", 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5000000000000001e-07 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106133+0000", 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8940000000000004e-06 2024-04-18T21:52:03.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106149+0000", 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5534000000000001e-05 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106159+0000", 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.047e-05 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418620+0000", 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31246021499999999 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418658+0000", 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8566000000000003e-05 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418736+0000", 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8157000000000004e-05 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.0352339820000003, 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39824586299999998, 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121692+0000", 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5499999999999997e-07 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121705+0000", 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3207000000000001e-05 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121718+0000", 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2913e-05 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121724+0000", 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5030000000000002e-06 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.519849+0000", 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39812503900000001 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.519937+0000", 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7346e-05 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.237 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:03.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:03.242 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:03.235+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:03.339 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:03.335+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:03.351 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.7 to 1 (10000) 2024-04-18T21:52:03.355 INFO:tasks.rados.rados.0.smithi097.stdout:137: done (1 left) 2024-04-18T21:52:03.355 INFO:tasks.rados.rados.0.smithi097.stdout:139: done (0 left) 2024-04-18T21:52:03.355 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 43 current snap is 8 2024-04-18T21:52:03.355 INFO:tasks.rados.rados.0.smithi097.stdout:140: seq_num 74 ranges {720896=475136,1196032=622592,1818624=147456} 2024-04-18T21:52:03.363 INFO:tasks.rados.rados.0.smithi097.stdout:140: writing smithi09731216-43 from 720896 to 1196032 tid 1 2024-04-18T21:52:03.367 INFO:tasks.rados.rados.0.smithi097.stdout:140: writing smithi09731216-43 from 1196032 to 1818624 tid 2 2024-04-18T21:52:03.368 INFO:tasks.rados.rados.0.smithi097.stdout:140: writing smithi09731216-43 from 1818624 to 1966080 tid 3 2024-04-18T21:52:03.369 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 10 current snap is 8 2024-04-18T21:52:03.369 INFO:tasks.rados.rados.0.smithi097.stdout:141: seq_num 75 ranges {0=491520,491520=770048} 2024-04-18T21:52:03.372 INFO:tasks.rados.rados.0.smithi097.stdout:141: writing smithi09731216-10 from 0 to 491520 tid 1 2024-04-18T21:52:03.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-18T21:52:03.378 INFO:tasks.rados.rados.0.smithi097.stdout:141: writing smithi09731216-10 from 491520 to 1261568 tid 2 2024-04-18T21:52:03.378 INFO:tasks.rados.rados.0.smithi097.stdout:142: snap_create 2024-04-18T21:52:03.408 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:03.408 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:03.408 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:03.408 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:03.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:03.440 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:03.439+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:03.541 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:03.538+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:03.589 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:03.589 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:03.589 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:03.589 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:03.589 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:03.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:03.606 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:03.603+0000 7f8621e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:03.642 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:03.638+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 29/26 ECSubWriteReply(tid=2, last_complete=29'3, committed=1, applied=1))", 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.684087592000001, 2024-04-18T21:52:03.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20866285400000001, 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034153+0000", 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3099999999999999e-07 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034154+0000", 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.838e-06 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034160+0000", 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3979999999999998e-06 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034179+0000", 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8751000000000001e-05 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242502+0000", 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.208323065 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242815+0000", 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031357100000000001 2024-04-18T21:52:03.722 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 29/26 ECSubWriteReply(tid=3, last_complete=29'4, committed=1, applied=1))", 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.684053382999998, 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20991810899999999, 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5400000000000002e-07 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034189+0000", 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.956e-06 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034196+0000", 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0509999999999999e-06 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034210+0000", 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4086e-05 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242874+0000", 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.208664241 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.244105+0000", 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001230521 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.fs1 30/26 ECSubWrite(tid=141, reqid=client.4509.0:88, at_version=30'3, trim_to=0'0, roll_forward_to=28'2))", 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.061043+0000", 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.657197153999999, 2024-04-18T21:52:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20224404100000001, 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061043+0000", 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061043+0000", 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061043+0000", 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5699999999999999e-07 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061314+0000", 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027134499999999998 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.061326+0000", 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1814000000000001e-05 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.161407+0000", 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10008054800000001 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.261648+0000", 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100241765 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.261663+0000", 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4545999999999999e-05 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.263249+0000", 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001585524 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.263287+0000", 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8342e-05 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.0s0 30/26 ECSubReadReply(tid=6, attrs_read=0))", 2024-04-18T21:52:03.730 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.345541+0000", 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.372699066999999, 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200690538, 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345541+0000", 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345541+0000", 2024-04-18T21:52:03.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345542+0000", 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3960000000000001e-06 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345567+0000", 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.472e-05 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.345590+0000", 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2690000000000001e-05 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.445667+0000", 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007769900000001 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.545898+0000", 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100231019 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.546231+0000", 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033301400000000002 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:94 3.0s0 3:033c99c3:::smithi09731216-32:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.162390155000001, 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40893576500000001, 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.734 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555853+0000", 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1889999999999999e-06 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556617+0000", 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076362200000000004 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556649+0000", 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2478999999999999e-05 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556665+0000", 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5837000000000001e-05 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556826+0000", 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016137200000000001 2024-04-18T21:52:03.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.557241+0000", 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00041440699999999999 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.559642+0000", 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024011430000000001 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563415+0000", 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003773233 2024-04-18T21:52:03.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964577+0000", 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40116224099999997 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964786+0000", 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000208242 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.739 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:95 3.0s0 3:033c99c3:::smithi09731216-32:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.161537058, 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858905099999998, 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 95 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:03.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556705+0000", 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6360000000000001e-06 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556715+0000", 2024-04-18T21:52:03.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.066e-05 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556734+0000", 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8323999999999999e-05 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556745+0000", 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1240000000000001e-05 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561327+0000", 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0045823330000000001 2024-04-18T21:52:03.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561583+0000", 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000255732 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.562607+0000", 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010240990000000001 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563867+0000", 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012605439999999999 2024-04-18T21:52:03.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965078+0000", 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40121052699999998 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965292+0000", 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021395600000000001 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.744 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:96 3.0s0 3:033c99c3:::smithi09731216-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.161439516000002, 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42323290800000002, 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 96 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:03.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556802+0000", 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7400000000000001e-06 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556808+0000", 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.993e-06 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556828+0000", 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9364e-05 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.656965+0000", 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10013702300000001 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762039+0000", 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105074903 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762350+0000", 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031089399999999998 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967780+0000", 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20542997700000001 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.972542+0000", 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004761932 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.979990+0000", 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0074475289999999996 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.980033+0000", 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3553000000000001e-05 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.4s2 30/26 ECSubWrite(tid=10, reqid=client.4509.0:97, at_version=30'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.860362+0000", 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.857878182, 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20550668799999999, 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860362+0000", 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860362+0000", 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860363+0000", 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9399999999999993e-07 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860513+0000", 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014979199999999999 2024-04-18T21:52:03.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860538+0000", 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5286000000000002e-05 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860552+0000", 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.385e-05 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.960935+0000", 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100383132 2024-04-18T21:52:03.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.960960+0000", 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5179000000000001e-05 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065868+0000", 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104908455 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.753 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.4s2 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.860641+0000", 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.857598734, 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.205319586, 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860641+0000", 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860641+0000", 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860642+0000", 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860660+0000", 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.859e-05 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860675+0000", 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5119999999999999e-05 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.860685+0000", 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9729999999999992e-06 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.062389+0000", 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201703938 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.062420+0000", 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0877000000000003e-05 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065961+0000", 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035406280000000001 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:222 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:03.757 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.705357927, 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30521668800000001, 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 222 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012885+0000", 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7760000000000002e-06 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012901+0000", 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6563e-05 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012934+0000", 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2719999999999998e-05 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012952+0000", 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7730000000000001e-05 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013179+0000", 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022670500000000001 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013200+0000", 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1466000000000001e-05 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114024+0000", 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100823616 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114179+0000", 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015544500000000001 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.318099+0000", 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203919667 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:223 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:03.762 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.705237498000001, 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40441434900000001, 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 223 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:03.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013004+0000", 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5400000000000001e-06 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013015+0000", 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0519999999999999e-05 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013077+0000", 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2848999999999994e-05 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013098+0000", 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1083000000000001e-05 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013359+0000", 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000260408 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013383+0000", 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3867e-05 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114847+0000", 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10146374399999999 2024-04-18T21:52:03.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114996+0000", 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149865 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.417417+0000", 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30242047300000002 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:224 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:03.767 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.705107406, 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40564824999999999, 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 224 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013134+0000", 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.387e-06 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013144+0000", 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0273e-05 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013166+0000", 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1857000000000001e-05 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013182+0000", 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5645999999999999e-05 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113542+0000", 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10035996599999999 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113552+0000", 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0492999999999999e-05 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215576+0000", 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102023869 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215730+0000", 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000153635 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.418781+0000", 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203051124 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:302 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:03.772 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6565890850000002, 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.406305586, 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 302 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061654+0000", 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061669+0000", 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4637000000000001e-05 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061699+0000", 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9791e-05 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061714+0000", 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5441999999999999e-05 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061811+0000", 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7153999999999998e-05 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061819+0000", 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9279999999999993e-06 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064181+0000", 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002361447 2024-04-18T21:52:03.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064306+0000", 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012514399999999999 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.065234+0000", 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092848599999999996 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.165102+0000", 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.099867808000000002 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467877+0000", 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.302775085 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467956+0000", 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9381000000000005e-05 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.778 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:303 3.0s0 3:00e42e84:::smithi09731216-41:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6564896950000003, 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50679023400000001, 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 303 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061751+0000", 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5499999999999998e-07 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061754+0000", 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2339999999999999e-06 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061763+0000", 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3950000000000007e-06 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061769+0000", 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9730000000000002e-06 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061904+0000", 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013470999999999999 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061911+0000", 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2200000000000003e-06 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163859+0000", 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101947416 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163977+0000", 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117971 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.164692+0000", 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071513499999999999 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267070+0000", 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102378067 2024-04-18T21:52:03.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568454+0000", 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.301384333 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568541+0000", 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6224999999999995e-05 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:304 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6564472229999998, 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60693648, 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 304 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:03.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3000000000000001e-07 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061797+0000", 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3680000000000002e-06 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061804+0000", 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0809999999999996e-06 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061809+0000", 2024-04-18T21:52:03.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5280000000000003e-06 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061950+0000", 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000140529 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061958+0000", 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5329999999999994e-06 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166016+0000", 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104057925 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166118+0000", 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010206500000000001 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166690+0000", 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057174600000000004 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267073+0000", 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100382727 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668597+0000", 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401524149 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668729+0000", 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132399 2024-04-18T21:52:03.790 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/31 ECSubWriteReply(tid=26, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.066455+0000", 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6517850709999999, 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200532874, 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.066455+0000", 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.066455+0000", 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.066456+0000", 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.234e-06 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.066461+0000", 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8359999999999999e-06 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.066480+0000", 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8790000000000001e-05 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166558+0000", 2024-04-18T21:52:03.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100078499 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.266971+0000", 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10041269 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.266988+0000", 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6824999999999999e-05 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/31 ECSubWriteReply(tid=28, last_complete=41'12, committed=1, applied=1))", 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.467918+0000", 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.2503217700000002, 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20173248299999999, 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467918+0000", 2024-04-18T21:52:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467918+0000", 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467918+0000", 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.97e-07 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467920+0000", 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6610000000000001e-06 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467924+0000", 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.574e-06 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467926+0000", 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.785e-06 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668450+0000", 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20052392899999999 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.669651+0000", 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012003370000000001 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:332 3.0s0 3:00b6e030:::smithi09731216-21:head [copy-from ver 39] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:03.798 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.5122882439999996, 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.72786893100000005, 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 332 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205954+0000", 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4370000000000001e-06 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205969+0000", 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4516e-05 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206001+0000", 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.252e-05 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206022+0000", 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.084e-05 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306220+0000", 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100197961 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306483+0000", 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026249200000000002 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.726594+0000", 2024-04-18T21:52:03.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42011143499999998 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.730261+0000", 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036666060000000002 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933714+0000", 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203453467 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933821+0000", 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010665700000000001 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:333 3.0s0 3:00b6e030:::smithi09731216-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.5121832709999996, 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52393701999999998, 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:03.804 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 333 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:03.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206058+0000", 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.426e-06 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206065+0000", 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4799999999999998e-06 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206085+0000", 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0089999999999999e-05 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206101+0000", 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.643e-05 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306839+0000", 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100738203 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306987+0000", 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014755599999999999 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729615+0000", 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42262781100000002 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729821+0000", 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020588799999999999 2024-04-18T21:52:03.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729994+0000", 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000173136 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.18s7 42/31 ECSubWrite(tid=42, reqid=client.4509.0:328, at_version=42'16, trim_to=0'0, roll_forward_to=38'13))", 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.413360+0000", 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.3048801220000001, 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20165618299999999, 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.809 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.413360+0000", 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.413360+0000", 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.413361+0000", 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8599999999999998e-07 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.413378+0000", 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7581999999999999e-05 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.413394+0000", 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5814000000000001e-05 2024-04-18T21:52:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.513488+0000", 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100094216 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.613761+0000", 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100273054 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.613799+0000", 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7693e-05 2024-04-18T21:52:03.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.614934+0000", 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011354889999999999 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.615016+0000", 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1649000000000004e-05 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.813 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:03.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:03.815 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:03.811+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:03.915 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:03.911+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:03.985 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:03.985 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:03.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.985 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDScrubReserve(3.3s6,RELEASE,e:47,from:4(0),reservation_nonce:0)", 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.916311+0000", 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.069292428000000003, 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069325813, 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.916311+0000", 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.916311+0000", 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.916312+0000", 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8700000000000004e-07 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.916318+0000", 2024-04-18T21:52:03.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2890000000000003e-06 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.916337+0000", 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9227000000000001e-05 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.916349+0000", 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1325000000000001e-05 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s6 47/45 ECSubWrite(tid=47, reqid=client.4509.0:365, at_version=47'11, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.928788+0000", 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.056815378, 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.056925626, 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928788+0000", 2024-04-18T21:52:03.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928788+0000", 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928789+0000", 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.062e-06 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928883+0000", 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4141000000000005e-05 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928900+0000", 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6662000000000001e-05 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928911+0000", 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1399e-05 2024-04-18T21:52:03.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929131+0000", 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000219662 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929149+0000", 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8039000000000001e-05 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.992 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s6 47/45 ECSubWrite(tid=48, reqid=client.4509.0:366, at_version=47'12, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.928932+0000", 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.056671048000000002, 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.056863143999999997, 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928932+0000", 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928932+0000", 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.928933+0000", 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3099999999999998e-07 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929018+0000", 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5143999999999997e-05 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929039+0000", 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1474000000000001e-05 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929053+0000", 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3969e-05 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929819+0000", 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076617199999999997 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929834+0000", 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4667e-05 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s6 47/45 ECSubWrite(tid=49, reqid=client.4509.0:367, at_version=47'13, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.929069+0000", 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.056534156000000002, 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.056803840000000001, 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.996 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929069+0000", 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929069+0000", 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929069+0000", 2024-04-18T21:52:03.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7399999999999999e-07 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929089+0000", 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9488999999999999e-05 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929097+0000", 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1410000000000005e-06 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929104+0000", 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1500000000000002e-06 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s6 47/45 ECSubWrite(tid=50, reqid=client.4509.0:361, at_version=47'14, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.929117+0000", 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.056486580000000002, 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.056819034999999997, 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:03.999 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929117+0000", 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929117+0000", 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929117+0000", 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9700000000000003e-07 2024-04-18T21:52:04.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929174+0000", 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6643e-05 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929194+0000", 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0545999999999999e-05 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.929203+0000", 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1789999999999997e-06 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s6 47/45 ECSubWrite(tid=51, reqid=client.4509.0:362, at_version=47'15, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.942686+0000", 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.042917645999999997, 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.043314345999999997, 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.002 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942686+0000", 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942686+0000", 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942687+0000", 2024-04-18T21:52:04.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0869999999999999e-06 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942750+0000", 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3175000000000004e-05 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942769+0000", 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9102000000000001e-05 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942779+0000", 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0268000000000001e-05 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s6 47/45 ECSubWrite(tid=52, reqid=client.4509.0:363, at_version=47'16, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.943869+0000", 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.04173404, 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.042193373999999999, 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.005 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943869+0000", 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943869+0000", 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943870+0000", 2024-04-18T21:52:04.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8000000000000004e-07 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943897+0000", 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6897e-05 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943915+0000", 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8284e-05 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943927+0000", 2024-04-18T21:52:04.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1551e-05 2024-04-18T21:52:04.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.008 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.008 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:04.008 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 7 2024-04-18T21:52:04.008 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:04.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:04.016 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:04.014+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:04.117 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:04.115+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:04.170 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:04.171 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:04.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:04.218 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:04.215+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:83 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 18.242812660999999, 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28825372900000001, 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.296 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 83 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2e-07 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050828+0000", 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3929999999999998e-06 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050832+0000", 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4909999999999999e-06 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050836+0000", 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5219999999999999e-06 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052455+0000", 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016189710000000001 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052552+0000", 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7213999999999995e-05 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052983+0000", 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043119300000000002 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138342+0000", 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.085359090999999998 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.338942+0000", 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200600308 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339078+0000", 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136026 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:84 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:04.301 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 18.242784003000001, 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59192865299999997, 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 84 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050854+0000", 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1399999999999998e-07 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050855+0000", 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5719999999999999e-06 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050860+0000", 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050864+0000", 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5750000000000001e-06 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138083+0000", 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087218981000000001 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138226+0000", 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000143545 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339757+0000", 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20153072 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339856+0000", 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9247999999999996e-05 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642782+0000", 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30292579400000003 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.es0 30/26 ECSubRead(tid=10, to_read={3:77cf8780:::smithi09731216-28:head=0,4096,0}, subchunks={3:77cf8780:::smithi09731216-28:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:04.306 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.95362343, 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218228800000002, 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340093+0000", 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8665000000000007e-05 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440291+0000", 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100198545 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641845+0000", 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20155352000000001 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642196+0000", 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035155800000000002 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.309 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 30/26 ECSubWriteReply(tid=9, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.952941306, 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20052165999999999, 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.310 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340697+0000", 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.004e-06 2024-04-18T21:52:04.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340703+0000", 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5260000000000001e-06 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340719+0000", 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.679e-05 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340729+0000", 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9429999999999995e-06 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541183+0000", 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200453357 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541218+0000", 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5040000000000003e-05 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340933+0000", 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.952704740000001, 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200445823, 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340933+0000", 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.314 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340933+0000", 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340933+0000", 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.61e-07 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340943+0000", 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4599999999999992e-06 2024-04-18T21:52:04.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340953+0000", 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.013e-05 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340958+0000", 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2939999999999996e-06 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541310+0000", 2024-04-18T21:52:04.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20035136200000001 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541378+0000", 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8816e-05 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.852783246000001, 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20088825699999999, 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.317 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440856+0000", 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9870000000000002e-06 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440895+0000", 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9325e-05 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440929+0000", 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3293000000000001e-05 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440950+0000", 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0812999999999999e-05 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641679+0000", 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20072974199999999 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641742+0000", 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3096999999999996e-05 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.320 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.852425360000002, 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20123138300000001, 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441213+0000", 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.057e-06 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441231+0000", 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7995999999999998e-05 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441251+0000", 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0032999999999999e-05 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441267+0000", 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5614e-05 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642333+0000", 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201065943 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642443+0000", 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011074 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.cs7 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.530082251, 2024-04-18T21:52:04.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30181939000000002, 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763557+0000", 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.776e-06 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763600+0000", 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3120000000000001e-05 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763635+0000", 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5199e-05 2024-04-18T21:52:04.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.863748+0000", 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100112399 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964004+0000", 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10025664500000001 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964030+0000", 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5299999999999998e-05 2024-04-18T21:52:04.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065244+0000", 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101214472 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065374+0000", 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000130479 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=19, last_complete=35'11, committed=1, applied=1))", 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 12.015426494, 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22495391000000001, 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278212+0000", 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.004e-06 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278218+0000", 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7490000000000004e-06 2024-04-18T21:52:04.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278233+0000", 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5815999999999999e-05 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378308+0000", 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007474600000001 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.502895+0000", 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.124586697 2024-04-18T21:52:04.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.503165+0000", 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026989800000000002 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=20, last_complete=35'12, committed=1, applied=1))", 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.915300460999999, 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225695955, 2024-04-18T21:52:04.332 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378338+0000", 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3499999999999996e-07 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378343+0000", 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0669999999999999e-06 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378356+0000", 2024-04-18T21:52:04.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3264e-05 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378367+0000", 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1069e-05 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.603689+0000", 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22532233900000001 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604033+0000", 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034358099999999998 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=21, last_complete=35'13, committed=1, applied=1))", 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.915260106, 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22690856100000001, 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.336 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378378+0000", 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378380+0000", 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7190000000000001e-06 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378389+0000", 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378395+0000", 2024-04-18T21:52:04.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5210000000000002e-06 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604185+0000", 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22579012100000001 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.605286+0000", 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011008979999999999 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 3.18s2,from:0'0,to:34'10,epoch:35/31,start:3:18000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378528+0000", 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.915109573000001, 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20032903399999999, 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378528+0000", 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378528+0000", 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378528+0000", 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2899999999999999e-07 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378533+0000", 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6229999999999996e-06 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378542+0000", 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.329e-06 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.478630+0000", 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10008789999999999 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.578810+0000", 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100180509 2024-04-18T21:52:04.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.578825+0000", 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4481000000000001e-05 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.578857+0000", 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1863000000000003e-05 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.343 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:230 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.280368757, 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.312981385, 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 230 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:04.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013270+0000", 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5300000000000002e-07 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013275+0000", 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013291+0000", 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5466000000000001e-05 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013303+0000", 2024-04-18T21:52:04.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2e-05 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013750+0000", 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044700300000000002 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013767+0000", 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7263999999999999e-05 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225043+0000", 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21127590299999999 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225199+0000", 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155716 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.326250+0000", 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10105134 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.348 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.bs3 36/26 ECSubWrite(tid=36, reqid=client.4509.0:216, at_version=36'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.275278712, 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20158620099999999, 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1299999999999999e-07 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018417+0000", 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7250000000000002e-05 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018446+0000", 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9567e-05 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118544+0000", 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10009759999999999 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118768+0000", 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022446200000000001 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118785+0000", 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6286e-05 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.219890+0000", 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10110504400000001 2024-04-18T21:52:04.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.219945+0000", 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5278999999999997e-05 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:315 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 6.2008606799999999, 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70415003200000004, 2024-04-18T21:52:04.353 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 315 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092778+0000", 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.725e-06 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092785+0000", 2024-04-18T21:52:04.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7950000000000002e-06 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092802+0000", 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6852999999999999e-05 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092810+0000", 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7950000000000008e-06 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152681+0000", 2024-04-18T21:52:04.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059871316000000001 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152755+0000", 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4179999999999998e-05 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153317+0000", 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00056207900000000005 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253434+0000", 2024-04-18T21:52:04.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100116332 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796715+0000", 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54328157899999996 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796927+0000", 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021137799999999999 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.358 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:316 3.es0 3:7b3c3586:::smithi09731216-12:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 6.200811936, 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70474535999999999, 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.359 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 316 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092826+0000", 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5999999999999999e-07 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092827+0000", 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714e-06 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092833+0000", 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0989999999999998e-06 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092836+0000", 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.056e-06 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153954+0000", 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061118211999999998 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154059+0000", 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010548399999999999 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154712+0000", 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065292300000000002 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253436+0000", 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098723985 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797355+0000", 2024-04-18T21:52:04.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.543919245 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797571+0000", 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021528199999999999 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:317 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 6.2007715330000002, 2024-04-18T21:52:04.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70519442399999999, 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 317 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000002e-07 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092868+0000", 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1440000000000001e-06 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092873+0000", 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0860000000000001e-06 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192949+0000", 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007585099999999 2024-04-18T21:52:04.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252838+0000", 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059889100000000001 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252927+0000", 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8651000000000002e-05 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253260+0000", 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033289600000000001 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.353758+0000", 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100498198 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797947+0000", 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.444189102 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.798060+0000", 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113109 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=6, reqid=client.4509.0:267, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.5881720369999996, 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.305911983, 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.370 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705468+0000", 2024-04-18T21:52:04.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4229999999999999e-06 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705683+0000", 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021499900000000001 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705719+0000", 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6758999999999998e-05 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805843+0000", 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10012386299999999 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806004+0000", 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160722 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806040+0000", 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6335999999999999e-05 2024-04-18T21:52:04.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011325+0000", 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20528459499999999 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011377+0000", 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2286e-05 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=8, reqid=client.4509.0:268, at_version=41'15, trim_to=0'0, roll_forward_to=41'12))", 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.4876489910000004, 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20598443299999999, 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805989+0000", 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5700000000000002e-07 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806154+0000", 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000165575 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806171+0000", 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6946e-05 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806183+0000", 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1394e-05 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.009717+0000", 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203534044 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.009751+0000", 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.455e-05 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011896+0000", 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002144721 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011973+0000", 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6645999999999998e-05 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=9, reqid=client.4509.0:269, at_version=41'16, trim_to=0'0, roll_forward_to=41'12))", 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.487435606, 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.206203308, 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.379 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.37e-07 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806231+0000", 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8955999999999999e-05 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806247+0000", 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5627000000000001e-05 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.906369+0000", 2024-04-18T21:52:04.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100122086 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011040+0000", 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104670897 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011064+0000", 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4219e-05 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.012353+0000", 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012883860000000001 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.012405+0000", 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2500000000000002e-05 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.383 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:04.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:04.385 INFO:tasks.rados.rados.0.smithi097.stdout:140: finishing write tid 1 to smithi09731216-43 2024-04-18T21:52:04.386 INFO:tasks.rados.rados.0.smithi097.stdout:141: finishing write tid 1 to smithi09731216-10 2024-04-18T21:52:04.386 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is set 2024-04-18T21:52:04.386 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.8 to 1 (10000) 2024-04-18T21:52:04.386 INFO:tasks.rados.rados.0.smithi097.stdout:142: done (2 left) 2024-04-18T21:52:04.387 INFO:tasks.rados.rados.0.smithi097.stdout:143: snap_create 2024-04-18T21:52:04.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-18T21:52:04.388 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:04.383+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:04.446 INFO:tasks.rados.rados.0.smithi097.stdout:141: finishing write tid 2 to smithi09731216-10 2024-04-18T21:52:04.485 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:04.482+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:04.542 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:04.542 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:361 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~622592 in=622592b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.17357768, 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.173602585, 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 361 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.543 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368928+0000", 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6559999999999999e-06 2024-04-18T21:52:04.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369217+0000", 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028880699999999997 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369238+0000", 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0914999999999999e-05 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369247+0000", 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1910000000000006e-06 2024-04-18T21:52:04.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369333+0000", 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5482000000000006e-05 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369345+0000", 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2147000000000001e-05 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395635+0000", 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026290322000000001 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395761+0000", 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126348 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837828+0000", 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44206649199999998 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837999+0000", 2024-04-18T21:52:04.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017102900000000001 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.840298+0000", 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022987670000000002 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944332+0000", 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10403438199999999 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.548 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:362 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~147456 in=147456b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.1725980540000001, 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1726971989999999, 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 362 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.549 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369907+0000", 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0529999999999999e-06 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369994+0000", 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6688999999999996e-05 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370003+0000", 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3789999999999995e-06 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370010+0000", 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9229999999999996e-06 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370098+0000", 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8146000000000006e-05 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370122+0000", 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3329e-05 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395809+0000", 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025687331000000001 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395945+0000", 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013612499999999999 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941837+0000", 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54589167500000002 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941940+0000", 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103435 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942633+0000", 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069312599999999996 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244973+0000", 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30233927599999999 2024-04-18T21:52:04.554 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:363 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.1724818800000001, 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1726391979999999, 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.555 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 363 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370023+0000", 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.75e-07 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370027+0000", 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8489999999999999e-06 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370039+0000", 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2013e-05 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370043+0000", 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6630000000000001e-06 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370169+0000", 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125506 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370180+0000", 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0798000000000001e-05 2024-04-18T21:52:04.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396197+0000", 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026017723999999999 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396320+0000", 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122905 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943149+0000", 2024-04-18T21:52:04.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54682909400000002 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943272+0000", 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012215699999999999 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943899+0000", 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000627106 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244975+0000", 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30107650600000002 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:364 3.11s0 3:8fad7925:::smithi09731216-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.370068+0000", 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.172435922, 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1726489950000001, 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for missing object", 2024-04-18T21:52:04.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 364 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370068+0000", 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370068+0000", 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370069+0000", 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3e-07 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370073+0000", 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0400000000000003e-06 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370080+0000", 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.311e-06 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370085+0000", 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.515e-06 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370225+0000", 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014028900000000001 2024-04-18T21:52:04.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370245+0000", 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9582999999999999e-05 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.496482+0000", 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.126237712 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.496611+0000", 2024-04-18T21:52:04.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012884999999999999 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:367 3.11s0 3:8af06b78:::smithi09731216-10:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.162748299, 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1630049250000001, 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.566 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 367 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4899999999999995e-07 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379760+0000", 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4170000000000001e-06 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379766+0000", 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7459999999999997e-06 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379769+0000", 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.608e-06 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379929+0000", 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015980200000000001 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379953+0000", 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3764000000000002e-05 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707668+0000", 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32771509500000001 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707855+0000", 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186987 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835388+0000", 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127533496 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835835+0000", 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044705299999999999 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.836767+0000", 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00093163299999999996 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944330+0000", 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107563336 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:368 3.11s0 3:8af06b78:::smithi09731216-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.1627123690000001, 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.163023817, 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for rw locks", 2024-04-18T21:52:04.572 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 368 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.91e-07 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379799+0000", 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2690000000000002e-06 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379805+0000", 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7989999999999999e-06 2024-04-18T21:52:04.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379809+0000", 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4929999999999998e-06 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379996+0000", 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186959 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.380015+0000", 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9837e-05 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707952+0000", 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32793653499999997 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.708113+0000", 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160536 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837592+0000", 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12947945499999999 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837760+0000", 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016796899999999999 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=50, last_complete=47'14, committed=1, applied=1))", 2024-04-18T21:52:04.577 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.31511539599999999, 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.315474421, 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8000000000000002e-07 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227392+0000", 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.57e-06 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227402+0000", 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9650000000000003e-06 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327516+0000", 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100114438 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=49, last_complete=47'13, committed=1, applied=1))", 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.234576+0000", 2024-04-18T21:52:04.580 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.30792781699999999, 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30831168599999997, 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234576+0000", 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234576+0000", 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234577+0000", 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0499999999999999e-07 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234580+0000", 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.072e-06 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234589+0000", 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0229999999999998e-06 2024-04-18T21:52:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234594+0000", 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3589999999999999e-06 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=50, last_complete=47'14, committed=1, applied=1))", 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.307901482, 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30830960200000002, 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.583 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:04.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5199999999999998e-07 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234605+0000", 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0229999999999999e-06 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234611+0000", 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9499999999999998e-06 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234615+0000", 2024-04-18T21:52:04.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2100000000000003e-06 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=51, last_complete=47'15, committed=1, applied=1))", 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.30653796700000002, 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30697058700000002, 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.586 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235967+0000", 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6899999999999998e-07 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235970+0000", 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3060000000000002e-06 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235981+0000", 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.083e-05 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235987+0000", 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9309999999999996e-06 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.588 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=52, last_complete=47'16, committed=1, applied=1))", 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.236587+0000", 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.30591695499999999, 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30637397100000002, 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236587+0000", 2024-04-18T21:52:04.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236587+0000", 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236588+0000", 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5300000000000004e-07 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236591+0000", 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.128e-06 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236601+0000", 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7890000000000006e-06 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236606+0000", 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6470000000000003e-06 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=51, last_complete=47'15, committed=1, applied=1))", 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.327557+0000", 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.21494749699999999, 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.215430977, 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327557+0000", 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327557+0000", 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327558+0000", 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6099999999999999e-07 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327565+0000", 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2729999999999997e-06 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327578+0000", 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3570999999999999e-05 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327585+0000", 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9759999999999998e-06 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.594 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=52, last_complete=47'16, committed=1, applied=1))", 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.214911357, 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.215419104, 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3099999999999999e-07 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327595+0000", 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8700000000000001e-06 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327600+0000", 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0010000000000001e-06 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327603+0000", 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5429999999999998e-06 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 13 2024-04-18T21:52:04.597 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:04.598 INFO:tasks.rados.rados.0.smithi097.stdout:141: finishing write tid 3 to smithi09731216-10 2024-04-18T21:52:04.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:04.600 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:04.594+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:04.647 INFO:tasks.rados.rados.0.smithi097.stdout:140: finishing write tid 2 to smithi09731216-43 2024-04-18T21:52:04.648 INFO:tasks.rados.rados.0.smithi097.stdout:140: finishing write tid 3 to smithi09731216-43 2024-04-18T21:52:04.699 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:04.695+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:04.712 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:04.713 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:04.713 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:04.713 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:04.713 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:04.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:04.749 INFO:tasks.rados.rados.0.smithi097.stdout:140: finishing write tid 4 to smithi09731216-43 2024-04-18T21:52:04.800 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:04.795+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:04.868 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:04.868 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:04.868 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.8s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 18.101160068999999, 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51395767000000003, 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.869 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.871 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.871 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764361+0000", 2024-04-18T21:52:04.871 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.61e-06 2024-04-18T21:52:04.871 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764368+0000", 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1600000000000001e-06 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764389+0000", 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1073e-05 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.874 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764409+0000", 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0006000000000001e-05 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866089+0000", 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101679558 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866221+0000", 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132022 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278264+0000", 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41204366100000001 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278317+0000", 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2580000000000001e-05 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.876 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:103 3.18s0 3:1b279e7b:::smithi09731216-35:head [append 0~573440,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.884501605000001, 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6106588159999999, 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 103 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981021+0000", 2024-04-18T21:52:04.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5470000000000001e-06 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981778+0000", 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00075675099999999995 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981811+0000", 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3200000000000001e-05 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981832+0000", 2024-04-18T21:52:04.879 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0703000000000002e-05 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.277997+0000", 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29616472700000002 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278003+0000", 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3269999999999997e-06 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591581+0000", 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.313578111 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591677+0000", 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5450000000000003e-05 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:104 3.18s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:04.881 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.883636751000001, 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714877378, 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.882 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981885+0000", 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0260000000000002e-06 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981895+0000", 2024-04-18T21:52:04.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7969999999999995e-06 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981918+0000", 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3116e-05 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981931+0000", 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2850000000000001e-05 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278086+0000", 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29615502199999999 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278088+0000", 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6230000000000001e-06 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696611+0000", 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418522515 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696760+0000", 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149429 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.883572444999999, 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9142408849999999, 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.886 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.887 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981948+0000", 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981952+0000", 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8739999999999997e-06 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981963+0000", 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.133e-05 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.888 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981971+0000", 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2900000000000002e-06 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278161+0000", 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29618945299999999 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278163+0000", 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.734e-06 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.889 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696819+0000", 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418656006 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.697058+0000", 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023910700000000001 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.896188+0000", 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19913024800000001 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.890 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:226 3.19s0 3:9941efaf:::smithi09731216-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013083+0000", 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.852436135, 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40562686100000001, 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 226 2024-04-18T21:52:04.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013083+0000", 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013083+0000", 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013085+0000", 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.53e-06 2024-04-18T21:52:04.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013095+0000", 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0319999999999999e-05 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013118+0000", 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2847e-05 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013136+0000", 2024-04-18T21:52:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7535999999999999e-05 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013367+0000", 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000231375 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013389+0000", 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1946999999999998e-05 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.894 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.213788+0000", 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20039871200000001 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.214027+0000", 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023972899999999999 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.418710+0000", 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20468286499999999 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.895 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:227 3.19s0 3:9941efaf:::smithi09731216-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.852360165, 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50662614500000003, 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 227 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.896 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013160+0000", 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1299999999999999e-07 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013167+0000", 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9670000000000004e-06 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013187+0000", 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9726000000000001e-05 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013202+0000", 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5483000000000001e-05 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.898 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013460+0000", 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025794499999999998 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013468+0000", 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8460000000000008e-06 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.315837+0000", 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30236893999999997 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.316131+0000", 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029429400000000001 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.519785+0000", 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20365413099999999 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:318 3.19s0 3:9da3af1e:::smithi09731216-23:head [delete] snapc 5=[2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 6.6726466230000003, 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60663671399999997, 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 318 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.901 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192874+0000", 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2699999999999998e-07 2024-04-18T21:52:04.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192878+0000", 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0239999999999999e-06 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192886+0000", 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2549999999999997e-06 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192890+0000", 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.357e-06 2024-04-18T21:52:04.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372031+0000", 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.179140625 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372092+0000", 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1457999999999993e-05 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372626+0000", 2024-04-18T21:52:04.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053349599999999995 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.374211+0000", 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001584752 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799295+0000", 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42508415500000002 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799510+0000", 2024-04-18T21:52:04.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021466500000000001 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:329 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.6418713499999997, 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60620628200000004, 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.906 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 329 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6600000000000003e-07 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223651+0000", 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5919999999999999e-06 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223655+0000", 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9890000000000003e-06 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223658+0000", 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2600000000000001e-06 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514422+0000", 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29076426599999999 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514584+0000", 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000161275 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.909 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625434+0000", 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.110850424 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625669+0000", 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234315 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.829854+0000", 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20418589500000001 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.910 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:360 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~475136] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.5013331999999999, 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98249101000000005, 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 360 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.911 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364189+0000", 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8420000000000001e-06 2024-04-18T21:52:04.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364486+0000", 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029724500000000002 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364517+0000", 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0994000000000002e-05 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364530+0000", 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2330999999999999e-05 2024-04-18T21:52:04.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364644+0000", 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114673 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364657+0000", 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2258e-05 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.394826+0000", 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.030169702 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395495+0000", 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00066827599999999998 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.821805+0000", 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42631051199999997 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.822025+0000", 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000220084 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824602+0000", 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025771510000000002 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944319+0000", 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11971646900000001 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346144+0000", 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401824764 2024-04-18T21:52:04.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346677+0000", 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053370900000000005 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:361 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~622592 in=622592b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.4965929689999999, 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2802270259999999, 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.917 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 361 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368928+0000", 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6559999999999999e-06 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369217+0000", 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028880699999999997 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369238+0000", 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0914999999999999e-05 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369247+0000", 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1910000000000006e-06 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369333+0000", 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5482000000000006e-05 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369345+0000", 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2147000000000001e-05 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395635+0000", 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026290322000000001 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395761+0000", 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126348 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837828+0000", 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44206649199999998 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837999+0000", 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017102900000000001 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.840298+0000", 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022987670000000002 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944332+0000", 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10403438199999999 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648919+0000", 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70458679599999996 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649154+0000", 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234692 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.923 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:362 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~147456 in=147456b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.495613343, 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.279510908, 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 362 2024-04-18T21:52:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369907+0000", 2024-04-18T21:52:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0529999999999999e-06 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369994+0000", 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6688999999999996e-05 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370003+0000", 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3789999999999995e-06 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370010+0000", 2024-04-18T21:52:04.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9229999999999996e-06 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370098+0000", 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8146000000000006e-05 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370122+0000", 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3329e-05 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395809+0000", 2024-04-18T21:52:04.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025687331000000001 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395945+0000", 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013612499999999999 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941837+0000", 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54589167500000002 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941940+0000", 2024-04-18T21:52:04.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103435 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942633+0000", 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069312599999999996 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244973+0000", 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30233927599999999 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649234+0000", 2024-04-18T21:52:04.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40426141100000001 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649417+0000", 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018301 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:363 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.4954971690000001, 2024-04-18T21:52:04.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.380372242, 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 363 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370023+0000", 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.75e-07 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370027+0000", 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8489999999999999e-06 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370039+0000", 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2013e-05 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370043+0000", 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6630000000000001e-06 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370169+0000", 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125506 2024-04-18T21:52:04.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370180+0000", 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0798000000000001e-05 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396197+0000", 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026017723999999999 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396320+0000", 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122905 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943149+0000", 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54682909400000002 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943272+0000", 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012215699999999999 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943899+0000", 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000627106 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244975+0000", 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30107650600000002 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.749791+0000", 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50481555700000003 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.750395+0000", 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060378899999999998 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:365 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~491520] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.4918874360000001, 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97351366699999997, 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 365 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:04.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373634+0000", 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.024e-06 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373873+0000", 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000238459 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373899+0000", 2024-04-18T21:52:04.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6461999999999998e-05 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373912+0000", 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2823000000000001e-05 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374028+0000", 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115758 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374050+0000", 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2314e-05 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.601683+0000", 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22763349599999999 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.602117+0000", 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043313999999999999 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828190+0000", 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226073089 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828481+0000", 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029162099999999999 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.830297+0000", 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018160229999999999 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944323+0000", 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114025979 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346866+0000", 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40254325000000002 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.347146+0000", 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027922899999999998 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:366 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~770048] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.486268505, 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.067907554, 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 366 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379253+0000", 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9609999999999999e-06 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379714+0000", 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046141699999999999 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379730+0000", 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5585000000000001e-05 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379738+0000", 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379830+0000", 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2551000000000002e-05 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379852+0000", 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.143e-05 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.604888+0000", 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225036559 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.605008+0000", 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119936 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831601+0000", 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226592508 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831778+0000", 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017783799999999999 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.834089+0000", 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023105930000000001 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944325+0000", 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11023567500000001 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.446970+0000", 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50264567000000004 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.447159+0000", 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018811199999999999 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:367 3.11s0 3:8af06b78:::smithi09731216-10:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.485763588, 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1678633460000001, 2024-04-18T21:52:04.950 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 367 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4899999999999995e-07 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379760+0000", 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4170000000000001e-06 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379766+0000", 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7459999999999997e-06 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379769+0000", 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.608e-06 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379929+0000", 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015980200000000001 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379953+0000", 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3764000000000002e-05 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707668+0000", 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32771509500000001 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707855+0000", 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186987 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835388+0000", 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127533496 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835835+0000", 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044705299999999999 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.836767+0000", 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00093163299999999996 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944330+0000", 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107563336 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547378+0000", 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60304798000000004 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547619+0000", 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024088000000000001 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.956 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=50, last_complete=47'14, committed=1, applied=1))", 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.63813068500000003, 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42177490000000001, 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8000000000000002e-07 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227392+0000", 2024-04-18T21:52:04.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.57e-06 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227402+0000", 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9650000000000003e-06 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327516+0000", 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100114438 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648849+0000", 2024-04-18T21:52:04.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32133336099999998 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649164+0000", 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031428599999999999 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=50, last_complete=47'14, committed=1, applied=1))", 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.63091677099999999, 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41408854499999997, 2024-04-18T21:52:04.960 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5199999999999998e-07 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234605+0000", 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0229999999999999e-06 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234611+0000", 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9499999999999998e-06 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234615+0000", 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2100000000000003e-06 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648672+0000", 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41405668699999998 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648691+0000", 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9423e-05 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=51, last_complete=47'15, committed=1, applied=1))", 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.62955325600000001, 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41281960600000001, 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235967+0000", 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6899999999999998e-07 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235970+0000", 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3060000000000002e-06 2024-04-18T21:52:04.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235981+0000", 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.083e-05 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235987+0000", 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9309999999999996e-06 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648743+0000", 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41275640400000002 2024-04-18T21:52:04.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648786+0000", 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2666000000000001e-05 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=52, last_complete=47'16, committed=1, applied=1))", 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.236587+0000", 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.62893224400000003, 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41223289699999999, 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236587+0000", 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236587+0000", 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236588+0000", 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5300000000000004e-07 2024-04-18T21:52:04.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236591+0000", 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.128e-06 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236601+0000", 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7890000000000006e-06 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.236606+0000", 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6470000000000003e-06 2024-04-18T21:52:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648803+0000", 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41219714200000002 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648820+0000", 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6637999999999999e-05 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=52, last_complete=47'16, committed=1, applied=1))", 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.53792664599999995, 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42417700800000002, 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3099999999999999e-07 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327595+0000", 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8700000000000001e-06 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327600+0000", 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0010000000000001e-06 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327603+0000", 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5429999999999998e-06 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.749596+0000", 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.421992433 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:04.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.751770+0000", 2024-04-18T21:52:04.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021739300000000001 2024-04-18T21:52:04.974 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.974 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.974 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.974 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:04.974 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:04.974 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:04.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:04.976 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:04.971+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:05.055 INFO:tasks.rados.rados.0.smithi097.stdout:141: finishing write tid 4 to smithi09731216-10 2024-04-18T21:52:05.055 INFO:tasks.rados.rados.0.smithi097.stdout:141: oid 10 updating version 0 to 11 2024-04-18T21:52:05.055 INFO:tasks.rados.rados.0.smithi097.stdout:141: oid 10 updating version 11 to 13 2024-04-18T21:52:05.055 INFO:tasks.rados.rados.0.smithi097.stdout:141: oid 10 version 13 is already newer than 12 2024-04-18T21:52:05.055 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 10 v 13 (ObjNum 75 snap 8 seq_num 75) dirty exists 2024-04-18T21:52:05.055 INFO:tasks.rados.rados.0.smithi097.stdout:141: left oid 10 (ObjNum 75 snap 8 seq_num 75) 2024-04-18T21:52:05.077 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:05.071+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:05.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:05.106 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:05.106 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:05.107 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:05.107 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:05.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:05.177 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:05.175+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:05.232 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:05.232 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:05.239 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:05.239 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:05.239 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:05.239 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:05.239 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:05.249 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:05.249 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:05.253 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:05.265 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:05.265 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:05.282 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:05.283 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:05.296 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:05.295+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:05.301 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:05.301 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:05.319 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:05.319 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:05.337 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:05.337 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:05.357 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:05.357 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.9 to 1 (10000) 2024-04-18T21:52:05.377 INFO:tasks.rados.rados.0.smithi097.stdout:141: done (2 left) 2024-04-18T21:52:05.377 INFO:tasks.rados.rados.0.smithi097.stdout:143: done (1 left) 2024-04-18T21:52:05.377 INFO:tasks.rados.rados.0.smithi097.stdout:144: read oid 8 snap -1 2024-04-18T21:52:05.377 INFO:tasks.rados.rados.0.smithi097.stdout:144: expect deleted 2024-04-18T21:52:05.377 INFO:tasks.rados.rados.0.smithi097.stdout:145: rmattr oid 41 current snap is 10 2024-04-18T21:52:05.377 INFO:tasks.rados.rados.0.smithi097.stdout:145: done (2 left) 2024-04-18T21:52:05.378 INFO:tasks.rados.rados.0.smithi097.stdout:146: setattr oid 10 current snap is 10 2024-04-18T21:52:05.378 INFO:tasks.rados.rados.0.smithi097.stdout:147: copy_from oid 24 from oid 20 current snap is 10 2024-04-18T21:52:05.378 INFO:tasks.rados.rados.0.smithi097.stdout:148: setattr oid 14 current snap is 10 2024-04-18T21:52:05.378 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:05.375+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:05.380 INFO:tasks.rados.rados.0.smithi097.stdout:149: snap_remove snap 5 2024-04-18T21:52:05.382 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:05.383 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:05.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-18T21:52:05.403 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:05.403 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:05.423 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:05.423 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:05.425 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:05.425 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:05.436 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:05.445 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:05.446 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:05.509 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:05.507+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:05.556 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:05.556 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:05.570 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:05.580 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:05.578+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:05.681 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:05.675+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:05.695 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:05.695 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:05.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:05.782 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:05.779+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:05.833 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:05.833 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:05.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:05.883 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:05.879+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:05.985 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:05.979+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:06.011 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:06.011 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:06.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:06.085 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:06.082+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:06.147 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:06.147 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:06.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:06.186 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:06.182+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:06.279 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:06.279 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:06.287 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:06.282+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:06.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:06.376 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.10 to 1 (10000) 2024-04-18T21:52:06.382 INFO:tasks.rados.rados.0.smithi097.stdout:149: done (5 left) 2024-04-18T21:52:06.383 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 30 current snap is 10 2024-04-18T21:52:06.383 INFO:tasks.rados.rados.0.smithi097.stdout:150: seq_num 76 ranges {0=507904,507904=507904,1015808=262144} 2024-04-18T21:52:06.386 INFO:tasks.rados.rados.0.smithi097.stdout:150: writing smithi09731216-30 from 0 to 507904 tid 1 2024-04-18T21:52:06.387 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:06.383+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:06.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-18T21:52:06.390 INFO:tasks.rados.rados.0.smithi097.stdout:150: writing smithi09731216-30 from 507904 to 1015808 tid 2 2024-04-18T21:52:06.392 INFO:tasks.rados.rados.0.smithi097.stdout:150: writing smithi09731216-30 from 1015808 to 1277952 tid 3 2024-04-18T21:52:06.392 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 5 current snap is 10 2024-04-18T21:52:06.392 INFO:tasks.rados.rados.0.smithi097.stdout:140: finishing write tid 5 to smithi09731216-43 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:140: oid 43 updating version 0 to 16 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:140: oid 43 version 16 is already newer than 15 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:140: oid 43 version 16 is already newer than 10 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:140: oid 43 version 16 is already newer than 14 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 43 v 16 (ObjNum 74 snap 8 seq_num 74) dirty exists 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:140: left oid 43 (ObjNum 74 snap 8 seq_num 74) 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 14 v 25 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:147: finishing copy_from racing read to smithi09731216-24 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:147: finishing copy_from to smithi09731216-24 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 24 v 39 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-04-18T21:52:06.393 INFO:tasks.rados.rados.0.smithi097.stdout:151: seq_num 77 ranges {81920=524288,606208=524288,1130496=245760} 2024-04-18T21:52:06.395 INFO:tasks.rados.rados.0.smithi097.stdout:151: writing smithi09731216-5 from 81920 to 606208 tid 1 2024-04-18T21:52:06.398 INFO:tasks.rados.rados.0.smithi097.stdout:151: writing smithi09731216-5 from 606208 to 1130496 tid 2 2024-04-18T21:52:06.400 INFO:tasks.rados.rados.0.smithi097.stdout:151: writing smithi09731216-5 from 1130496 to 1376256 tid 3 2024-04-18T21:52:06.400 INFO:tasks.rados.rados.0.smithi097.stdout:140: done (6 left) 2024-04-18T21:52:06.400 INFO:tasks.rados.rados.0.smithi097.stdout:144: done (5 left) 2024-04-18T21:52:06.400 INFO:tasks.rados.rados.0.smithi097.stdout:147: done (4 left) 2024-04-18T21:52:06.400 INFO:tasks.rados.rados.0.smithi097.stdout:148: done (3 left) 2024-04-18T21:52:06.401 INFO:tasks.rados.rados.0.smithi097.stdout:152: snap_remove snap 0 2024-04-18T21:52:06.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-18T21:52:06.416 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:06.416 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:06.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:06.517 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:06.515+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:06.589 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:06.587+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:06.597 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:06.597 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:06.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:06.691 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:06.690+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:06.780 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:06.780 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:06.791 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:06.790+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:06.791 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:06.790+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:06.791 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:06.787+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:06.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:06.892 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:06.891+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:06.929 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:06.929 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:06.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:06.993 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:06.987+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:07.063 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:07.063 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:07.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:07.093 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:07.091+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:07.195 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:07.191+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:07.201 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:07.201 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:07.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:07.295 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:07.291+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:07.338 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:07.338 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:07.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is set 2024-04-18T21:52:07.353 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.11 to 1 (10000) 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:152: done (3 left) 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:153: delete oid 49 current snap is 10 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:150: finishing write tid 1 to smithi09731216-30 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:150: finishing write tid 2 to smithi09731216-30 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:150: finishing write tid 3 to smithi09731216-30 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:150: finishing write tid 4 to smithi09731216-30 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 10 v 18 (ObjNum 75 snap 8 seq_num 75) dirty exists 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:151: finishing write tid 1 to smithi09731216-5 2024-04-18T21:52:07.358 INFO:tasks.rados.rados.0.smithi097.stdout:150: finishing write tid 5 to smithi09731216-30 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:150: oid 30 updating version 0 to 9 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:150: oid 30 version 9 is already newer than 8 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:150: oid 30 updating version 9 to 10 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:150: oid 30 updating version 10 to 11 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 30 v 11 (ObjNum 76 snap 10 seq_num 76) dirty exists 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:150: left oid 30 (ObjNum 76 snap 10 seq_num 76) 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:151: finishing write tid 2 to smithi09731216-5 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:151: finishing write tid 3 to smithi09731216-5 2024-04-18T21:52:07.359 INFO:tasks.rados.rados.0.smithi097.stdout:151: finishing write tid 4 to smithi09731216-5 2024-04-18T21:52:07.388 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-18T21:52:07.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:07.399 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:07.398+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:07.466 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:07.480 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:07.481 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:07.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:07.529 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:07.527+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:07.561 INFO:tasks.rados.rados.0.smithi097.stdout:151: finishing write tid 5 to smithi09731216-5 2024-04-18T21:52:07.561 INFO:tasks.rados.rados.0.smithi097.stdout:151: oid 5 updating version 0 to 13 2024-04-18T21:52:07.561 INFO:tasks.rados.rados.0.smithi097.stdout:151: oid 5 updating version 13 to 14 2024-04-18T21:52:07.561 INFO:tasks.rados.rados.0.smithi097.stdout:151: oid 5 version 14 is already newer than 11 2024-04-18T21:52:07.561 INFO:tasks.rados.rados.0.smithi097.stdout:151: oid 5 version 14 is already newer than 12 2024-04-18T21:52:07.561 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 5 v 14 (ObjNum 77 snap 10 seq_num 77) dirty exists 2024-04-18T21:52:07.561 INFO:tasks.rados.rados.0.smithi097.stdout:151: left oid 5 (ObjNum 77 snap 10 seq_num 77) 2024-04-18T21:52:07.598 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:07.595+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:07.621 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:07.621 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:07.636 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:07.637 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:07.652 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:07.652 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:07.659 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:07.659 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:07.669 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:07.670 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:07.675 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:07.698 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:07.695+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:07.748 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:07.749 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:07.766 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-04-18T21:52:07.354307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":45384224,"num_objects":90,"num_object_clones":36,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":6,"num_read":230,"num_read_kb":14148,"num_write":299,"num_write_kb":25018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":231,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":441148,"kb_used_data":91860,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1132020932,"statfs":{"total":1159641169920,"available":1159189434368,"internally_reserved":0,"allocated":94064640,"data_stored":110481538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":357414205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1,16,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5611520,"num_objects":9,"num_object_clones":5,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":-75,"num_read_kb":-9289,"num_write":-112,"num_write_kb":-14488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000270"},"pg_stats":[{"pgid":"3.19","version":"41'9","reported_seq":121,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.853140+0000","last_change":"2024-04-18T21:52:01.767047+0000","last_active":"2024-04-18T21:52:04.853140+0000","last_peered":"2024-04-18T21:52:04.853140+0000","last_clean":"2024-04-18T21:52:04.853140+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:04.853140+0000","last_undegraded":"2024-04-18T21:52:04.853140+0000","last_fullsized":"2024-04-18T21:52:04.853140+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:12.682126+0000","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0.102052106,"stat_sum":{"num_bytes":262144,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.18","version":"42'16","reported_seq":123,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.750839+0000","last_change":"2024-04-18T21:52:00.261138+0000","last_active":"2024-04-18T21:52:04.750839+0000","last_peered":"2024-04-18T21:52:04.750839+0000","last_clean":"2024-04-18T21:52:04.750839+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:04.750839+0000","last_undegraded":"2024-04-18T21:52:04.750839+0000","last_fullsized":"2024-04-18T21:52:04.750839+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.847030+0000","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0.0011527200000000001,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.17","version":"39'6","reported_seq":108,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.502201+0000","last_change":"2024-04-18T21:51:59.786767+0000","last_active":"2024-04-18T21:52:03.502201+0000","last_peered":"2024-04-18T21:52:03.502201+0000","last_clean":"2024-04-18T21:52:03.502201+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:03.502201+0000","last_undegraded":"2024-04-18T21:52:03.502201+0000","last_fullsized":"2024-04-18T21:52:03.502201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.350212+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.10179099799999999,"stat_sum":{"num_bytes":671744,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.16","version":"32'3","reported_seq":109,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401291+0000","last_change":"2024-04-18T21:52:01.391849+0000","last_active":"2024-04-18T21:52:03.401291+0000","last_peered":"2024-04-18T21:52:03.401291+0000","last_clean":"2024-04-18T21:52:03.401291+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:03.401291+0000","last_undegraded":"2024-04-18T21:52:03.401291+0000","last_fullsized":"2024-04-18T21:52:03.401291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.764499+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.0012794480000000001,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.15","version":"39'7","reported_seq":113,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.757356+0000","last_change":"2024-04-18T21:51:58.257029+0000","last_active":"2024-04-18T21:52:04.757356+0000","last_peered":"2024-04-18T21:52:04.757356+0000","last_clean":"2024-04-18T21:52:04.757356+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:04.757356+0000","last_undegraded":"2024-04-18T21:52:04.757356+0000","last_fullsized":"2024-04-18T21:52:04.757356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.979554+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.0019999430000000001,"stat_sum":{"num_bytes":557056,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.14","version":"38'6","reported_seq":112,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383666+0000","last_change":"2024-04-18T21:51:58.380525+0000","last_active":"2024-04-18T21:52:05.383666+0000","last_peered":"2024-04-18T21:52:05.383666+0000","last_clean":"2024-04-18T21:52:05.383666+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:05.383666+0000","last_undegraded":"2024-04-18T21:52:05.383666+0000","last_fullsized":"2024-04-18T21:52:05.383666+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.480130+0000","scrub_duration":1515,"objects_trimmed":0,"snaptrim_duration":0.212985007,"stat_sum":{"num_bytes":770048,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.650306+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:05.650306+0000","last_peered":"2024-04-18T21:52:05.650306+0000","last_clean":"2024-04-18T21:52:05.650306+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:05.650306+0000","last_undegraded":"2024-04-18T21:52:05.650306+0000","last_fullsized":"2024-04-18T21:52:05.650306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"32'2","reported_seq":105,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401570+0000","last_change":"2024-04-18T21:52:02.163455+0000","last_active":"2024-04-18T21:52:03.401570+0000","last_peered":"2024-04-18T21:52:03.401570+0000","last_clean":"2024-04-18T21:52:03.401570+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:03.401570+0000","last_undegraded":"2024-04-18T21:52:03.401570+0000","last_fullsized":"2024-04-18T21:52:03.401570+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.309515+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.0014775280000000001,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.503144+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:03.503144+0000","last_peered":"2024-04-18T21:52:03.503144+0000","last_clean":"2024-04-18T21:52:03.503144+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:03.503144+0000","last_undegraded":"2024-04-18T21:52:03.503144+0000","last_fullsized":"2024-04-18T21:52:03.503144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":99,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.502694+0000","last_change":"2024-04-18T21:51:58.176158+0000","last_active":"2024-04-18T21:52:03.502694+0000","last_peered":"2024-04-18T21:52:03.502694+0000","last_clean":"2024-04-18T21:52:03.502694+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:03.502694+0000","last_undegraded":"2024-04-18T21:52:03.502694+0000","last_fullsized":"2024-04-18T21:52:03.502694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:18.918419+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.0019225869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.650707+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:04.650707+0000","last_peered":"2024-04-18T21:52:04.650707+0000","last_clean":"2024-04-18T21:52:04.650707+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:04.650707+0000","last_undegraded":"2024-04-18T21:52:04.650707+0000","last_fullsized":"2024-04-18T21:52:04.650707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":111,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.384380+0000","last_change":"2024-04-18T21:51:58.171632+0000","last_active":"2024-04-18T21:52:05.384380+0000","last_peered":"2024-04-18T21:52:05.384380+0000","last_clean":"2024-04-18T21:52:05.384380+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:05.384380+0000","last_undegraded":"2024-04-18T21:52:05.384380+0000","last_fullsized":"2024-04-18T21:52:05.384380+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:12.138527+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.10378514599999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1585,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401143+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:03.401143+0000","last_peered":"2024-04-18T21:52:03.401143+0000","last_clean":"2024-04-18T21:52:03.401143+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:03.401143+0000","last_undegraded":"2024-04-18T21:52:03.401143+0000","last_fullsized":"2024-04-18T21:52:03.401143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"33'7","reported_seq":112,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.754684+0000","last_change":"2024-04-18T21:51:58.157867+0000","last_active":"2024-04-18T21:52:04.754684+0000","last_peered":"2024-04-18T21:52:04.754684+0000","last_clean":"2024-04-18T21:52:04.754684+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:04.754684+0000","last_undegraded":"2024-04-18T21:52:04.754684+0000","last_fullsized":"2024-04-18T21:52:04.754684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:19.858344+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.002328845,"stat_sum":{"num_bytes":835584,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.3","version":"45'58","reported_seq":186,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.450274+0000","last_change":"2024-04-18T21:52:03.915511+0000","last_active":"2024-04-18T21:52:05.450274+0000","last_peered":"2024-04-18T21:52:05.450274+0000","last_clean":"2024-04-18T21:52:05.450274+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:05.450274+0000","last_undegraded":"2024-04-18T21:52:05.450274+0000","last_fullsized":"2024-04-18T21:52:05.450274+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:20.101521+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.30771508800000003,"stat_sum":{"num_bytes":8060928,"num_objects":16,"num_object_clones":8,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.756281+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:04.756281+0000","last_peered":"2024-04-18T21:52:04.756281+0000","last_clean":"2024-04-18T21:52:04.756281+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:04.756281+0000","last_undegraded":"2024-04-18T21:52:04.756281+0000","last_fullsized":"2024-04-18T21:52:04.756281+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"42'14","reported_seq":133,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.449482+0000","last_change":"2024-04-18T21:52:01.703639+0000","last_active":"2024-04-18T21:52:05.449482+0000","last_peered":"2024-04-18T21:52:05.449482+0000","last_clean":"2024-04-18T21:52:05.449482+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:05.449482+0000","last_undegraded":"2024-04-18T21:52:05.449482+0000","last_fullsized":"2024-04-18T21:52:05.449482+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.919850+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.204535667,"stat_sum":{"num_bytes":2883584,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.599956+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:03.599956+0000","last_peered":"2024-04-18T21:52:03.599956+0000","last_clean":"2024-04-18T21:52:03.599956+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:03.599956+0000","last_undegraded":"2024-04-18T21:52:03.599956+0000","last_fullsized":"2024-04-18T21:52:03.599956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"34'8","reported_seq":115,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.400694+0000","last_change":"2024-04-18T21:51:58.075957+0000","last_active":"2024-04-18T21:52:03.400694+0000","last_peered":"2024-04-18T21:52:03.400694+0000","last_clean":"2024-04-18T21:52:03.400694+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:03.400694+0000","last_undegraded":"2024-04-18T21:52:03.400694+0000","last_fullsized":"2024-04-18T21:52:03.400694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:07.467667+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0.001275502,"stat_sum":{"num_bytes":1458176,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.463708+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:04.463708+0000","last_peered":"2024-04-18T21:52:04.463708+0000","last_clean":"2024-04-18T21:52:04.463708+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:04.463708+0000","last_undegraded":"2024-04-18T21:52:04.463708+0000","last_fullsized":"2024-04-18T21:52:04.463708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"29'2","reported_seq":113,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383155+0000","last_change":"2024-04-18T21:52:02.588926+0000","last_active":"2024-04-18T21:52:05.383155+0000","last_peered":"2024-04-18T21:52:05.383155+0000","last_clean":"2024-04-18T21:52:05.383155+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:05.383155+0000","last_undegraded":"2024-04-18T21:52:05.383155+0000","last_fullsized":"2024-04-18T21:52:05.383155+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.058469+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.10213271,"stat_sum":{"num_bytes":212992,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.757118+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:04.757118+0000","last_peered":"2024-04-18T21:52:04.757118+0000","last_clean":"2024-04-18T21:52:04.757118+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:04.757118+0000","last_undegraded":"2024-04-18T21:52:04.757118+0000","last_fullsized":"2024-04-18T21:52:04.757118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":241,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.363921+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:04.363921+0000","last_peered":"2024-04-18T21:52:04.363921+0000","last_clean":"2024-04-18T21:52:04.363921+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:04.363921+0000","last_undegraded":"2024-04-18T21:52:04.363921+0000","last_fullsized":"2024-04-18T21:52:04.363921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":104,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.750539+0000","last_change":"2024-04-18T21:52:00.379115+0000","last_active":"2024-04-18T21:52:04.750539+0000","last_peered":"2024-04-18T21:52:04.750539+0000","last_clean":"2024-04-18T21:52:04.750539+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:04.750539+0000","last_undegraded":"2024-04-18T21:52:04.750539+0000","last_fullsized":"2024-04-18T21:52:04.750539+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.029798+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.00080647200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.9","version":"29'6","reported_seq":116,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.752259+0000","last_change":"2024-04-18T21:52:02.588756+0000","last_active":"2024-04-18T21:52:04.752259+0000","last_peered":"2024-04-18T21:52:04.752259+0000","last_clean":"2024-04-18T21:52:04.752259+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:04.752259+0000","last_undegraded":"2024-04-18T21:52:04.752259+0000","last_fullsized":"2024-04-18T21:52:04.752259+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:32.361581+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.205802136,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.a","version":"41'11","reported_seq":102,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.550641+0000","last_change":"2024-04-18T21:51:58.787646+0000","last_active":"2024-04-18T21:52:05.550641+0000","last_peered":"2024-04-18T21:52:05.550641+0000","last_clean":"2024-04-18T21:52:05.550641+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:05.550641+0000","last_undegraded":"2024-04-18T21:52:05.550641+0000","last_fullsized":"2024-04-18T21:52:05.550641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:45.279750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51759911599999997,"stat_sum":{"num_bytes":1261568,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.b","version":"49'32","reported_seq":122,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.700061+0000","last_change":"2024-04-18T21:51:58.887472+0000","last_active":"2024-04-18T21:52:05.700061+0000","last_peered":"2024-04-18T21:52:05.700061+0000","last_clean":"2024-04-18T21:52:05.700061+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:05.700061+0000","last_undegraded":"2024-04-18T21:52:05.700061+0000","last_fullsized":"2024-04-18T21:52:05.700061+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:50.009470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.72128645800000002,"stat_sum":{"num_bytes":5472256,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.c","version":"41'23","reported_seq":117,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.756712+0000","last_change":"2024-04-18T21:51:58.468330+0000","last_active":"2024-04-18T21:52:04.756712+0000","last_peered":"2024-04-18T21:52:04.756712+0000","last_clean":"2024-04-18T21:52:04.756712+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:04.756712+0000","last_undegraded":"2024-04-18T21:52:04.756712+0000","last_fullsized":"2024-04-18T21:52:04.756712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.442380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21306905500000001,"stat_sum":{"num_bytes":3915776,"num_objects":9,"num_object_clones":4,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":22,"num_read_kb":2008,"num_write":36,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":84,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.364507+0000","last_change":"2024-04-18T21:51:58.155757+0000","last_active":"2024-04-18T21:52:04.364507+0000","last_peered":"2024-04-18T21:52:04.364507+0000","last_clean":"2024-04-18T21:52:04.364507+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:04.364507+0000","last_undegraded":"2024-04-18T21:52:04.364507+0000","last_fullsized":"2024-04-18T21:52:04.364507+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:04.424721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00143743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.e","version":"43'24","reported_seq":118,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.364670+0000","last_change":"2024-04-18T21:51:58.155377+0000","last_active":"2024-04-18T21:52:04.364670+0000","last_peered":"2024-04-18T21:52:04.364670+0000","last_clean":"2024-04-18T21:52:04.364670+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:04.364670+0000","last_undegraded":"2024-04-18T21:52:04.364670+0000","last_fullsized":"2024-04-18T21:52:04.364670+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.054741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002880685,"stat_sum":{"num_bytes":2785280,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":4039,"num_write":17,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.f","version":"41'7","reported_seq":98,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.365041+0000","last_change":"2024-04-18T21:51:58.155833+0000","last_active":"2024-04-18T21:52:04.365041+0000","last_peered":"2024-04-18T21:52:04.365041+0000","last_clean":"2024-04-18T21:52:04.365041+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:04.365041+0000","last_undegraded":"2024-04-18T21:52:04.365041+0000","last_fullsized":"2024-04-18T21:52:04.365041+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:08.333087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014725999999999999,"stat_sum":{"num_bytes":491520,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":2884,"num_write":6,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.10","version":"41'16","reported_seq":89,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.550065+0000","last_change":"2024-04-18T21:52:03.761892+0000","last_active":"2024-04-18T21:52:05.550065+0000","last_peered":"2024-04-18T21:52:05.550065+0000","last_clean":"2024-04-18T21:52:05.550065+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:05.550065+0000","last_undegraded":"2024-04-18T21:52:05.550065+0000","last_fullsized":"2024-04-18T21:52:05.550065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:16.187800+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0.103603797,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.11","version":"47'16","reported_seq":47,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-18T21:52:05.055442+0000","last_change":"2024-04-18T21:52:04.346146+0000","last_active":"2024-04-18T21:52:05.055442+0000","last_peered":"2024-04-18T21:52:05.055442+0000","last_clean":"2024-04-18T21:51:50.243799+0000","last_became_active":"2024-04-18T21:52:02.703977+0000","last_became_peered":"2024-04-18T21:52:02.703977+0000","last_unstale":"2024-04-18T21:52:05.055442+0000","last_undegraded":"2024-04-18T21:52:05.055442+0000","last_fullsized":"2024-04-18T21:51:50.243799+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3235840,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":3320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,2147483647,1,5,2,2147483647],"avail_no_missing":["3(0)","1(4)","2(5)","2(6)","3(2)","4(1)","5(4)","5(5)","6(2)","6(6)","7(7)","8(7)","9(3)","10(3)","11(0)"],"object_location_counts":[{"shards":"1(4),2(6),3(0),4(1),5(5),6(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.12","version":"33'5","reported_seq":114,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383126+0000","last_change":"2024-04-18T21:52:01.496556+0000","last_active":"2024-04-18T21:52:05.383126+0000","last_peered":"2024-04-18T21:52:05.383126+0000","last_clean":"2024-04-18T21:52:05.383126+0000","last_became_active":"2024-04-18T21:51:48.492285+0000","last_became_peered":"2024-04-18T21:51:48.492285+0000","last_unstale":"2024-04-18T21:52:05.383126+0000","last_undegraded":"2024-04-18T21:52:05.383126+0000","last_fullsized":"2024-04-18T21:52:05.383126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'5","last_scrub_stamp":"2024-04-18T21:52:01.496294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.496294+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.628661+0000","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0.10167762499999999,"stat_sum":{"num_bytes":819200,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.13","version":"45'22","reported_seq":134,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.449983+0000","last_change":"2024-04-18T21:52:02.183412+0000","last_active":"2024-04-18T21:52:05.449983+0000","last_peered":"2024-04-18T21:52:05.449983+0000","last_clean":"2024-04-18T21:52:05.449983+0000","last_became_active":"2024-04-18T21:51:48.486639+0000","last_became_peered":"2024-04-18T21:51:48.486639+0000","last_unstale":"2024-04-18T21:52:05.449983+0000","last_undegraded":"2024-04-18T21:52:05.449983+0000","last_fullsized":"2024-04-18T21:52:05.449983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21032728000000001,"stat_sum":{"num_bytes":3137536,"num_objects":9,"num_object_clones":3,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":20,"num_read_kb":2981,"num_write":15,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":44793856,"num_objects":88,"num_object_clones":36,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":6,"num_read":104,"num_read_kb":14039,"num_write":118,"num_write_kb":22080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87359488,"data_stored":106635264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378635,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37648,"kb_used_data":6468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334192,"statfs":{"total":96636764160,"available":96598212608,"internally_reserved":0,"allocated":6623232,"data_stored":8016391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378635,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40332,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331508,"statfs":{"total":96636764160,"available":96595464192,"internally_reserved":0,"allocated":9359360,"data_stored":10891585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378636,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39940,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331900,"statfs":{"total":96636764160,"available":96595865600,"internally_reserved":0,"allocated":8970240,"data_stored":10772999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378635,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40604,"kb_used_data":9412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331236,"statfs":{"total":96636764160,"available":96595185664,"internally_reserved":0,"allocated":9637888,"data_stored":11514177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378635,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39132,"kb_used_data":7944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332708,"statfs":{"total":96636764160,"available":96596692992,"internally_reserved":0,"allocated":8134656,"data_stored":9728099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,16,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411338,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":4972544,"data_stored":5701731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32600,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":5701632,"data_stored":6546273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411338,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40220,"kb_used_data":9040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331620,"statfs":{"total":96636764160,"available":96595578880,"internally_reserved":0,"allocated":9256960,"data_stored":10944833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411339,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37276,"kb_used_data":10244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334564,"statfs":{"total":96636764160,"available":96598593536,"internally_reserved":0,"allocated":10489856,"data_stored":12472641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411337,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34452,"kb_used_data":7424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337388,"statfs":{"total":96636764160,"available":96601485312,"internally_reserved":0,"allocated":7602176,"data_stored":8884867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444042,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34140,"kb_used_data":7112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337700,"statfs":{"total":96636764160,"available":96601804800,"internally_reserved":0,"allocated":7282688,"data_stored":8233059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444042,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":6033408,"data_stored":6774883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6553600,"data_stored":8073216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9441280,"data_stored":11612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9961472,"data_stored":12181504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7618560,"data_stored":9445376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8429568,"data_stored":10473472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8728576,"data_stored":10653696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":10989568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4640768,"data_stored":5726208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5517312,"data_stored":6492160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4456448,"data_stored":5419008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6082560,"data_stored":7618560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6766592,"data_stored":7950336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:07.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:07.799 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:07.794+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:07.799 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:07.799 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:07.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:07.863 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:07.858+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:07.863 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:07.858+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:07.899 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:07.898+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:07.900 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:07.900 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:07.936 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:07.936 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:07.978 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:07.978 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:07.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:08.000 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:07.995+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:08.064 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:08.059+0000 7f8625ea1640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 2 after trimming was started 2024-04-18T21:52:08.101 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:08.098+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:08.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:08.137 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:08.155 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-04-18T21:52:07.354307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":45384224,"num_objects":90,"num_object_clones":36,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":6,"num_read":230,"num_read_kb":14148,"num_write":299,"num_write_kb":25018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":231,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":441148,"kb_used_data":91860,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1132020932,"statfs":{"total":1159641169920,"available":1159189434368,"internally_reserved":0,"allocated":94064640,"data_stored":110481538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":357414205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1,16,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5611520,"num_objects":9,"num_object_clones":5,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":-75,"num_read_kb":-9289,"num_write":-112,"num_write_kb":-14488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000270"},"pg_stats":[{"pgid":"3.19","version":"41'9","reported_seq":121,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.853140+0000","last_change":"2024-04-18T21:52:01.767047+0000","last_active":"2024-04-18T21:52:04.853140+0000","last_peered":"2024-04-18T21:52:04.853140+0000","last_clean":"2024-04-18T21:52:04.853140+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:04.853140+0000","last_undegraded":"2024-04-18T21:52:04.853140+0000","last_fullsized":"2024-04-18T21:52:04.853140+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:12.682126+0000","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0.102052106,"stat_sum":{"num_bytes":262144,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.18","version":"42'16","reported_seq":123,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.750839+0000","last_change":"2024-04-18T21:52:00.261138+0000","last_active":"2024-04-18T21:52:04.750839+0000","last_peered":"2024-04-18T21:52:04.750839+0000","last_clean":"2024-04-18T21:52:04.750839+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:04.750839+0000","last_undegraded":"2024-04-18T21:52:04.750839+0000","last_fullsized":"2024-04-18T21:52:04.750839+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.847030+0000","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0.0011527200000000001,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.17","version":"39'6","reported_seq":108,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.502201+0000","last_change":"2024-04-18T21:51:59.786767+0000","last_active":"2024-04-18T21:52:03.502201+0000","last_peered":"2024-04-18T21:52:03.502201+0000","last_clean":"2024-04-18T21:52:03.502201+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:03.502201+0000","last_undegraded":"2024-04-18T21:52:03.502201+0000","last_fullsized":"2024-04-18T21:52:03.502201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.350212+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.10179099799999999,"stat_sum":{"num_bytes":671744,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.16","version":"32'3","reported_seq":109,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401291+0000","last_change":"2024-04-18T21:52:01.391849+0000","last_active":"2024-04-18T21:52:03.401291+0000","last_peered":"2024-04-18T21:52:03.401291+0000","last_clean":"2024-04-18T21:52:03.401291+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:03.401291+0000","last_undegraded":"2024-04-18T21:52:03.401291+0000","last_fullsized":"2024-04-18T21:52:03.401291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.764499+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.0012794480000000001,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.15","version":"39'7","reported_seq":113,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.757356+0000","last_change":"2024-04-18T21:51:58.257029+0000","last_active":"2024-04-18T21:52:04.757356+0000","last_peered":"2024-04-18T21:52:04.757356+0000","last_clean":"2024-04-18T21:52:04.757356+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:04.757356+0000","last_undegraded":"2024-04-18T21:52:04.757356+0000","last_fullsized":"2024-04-18T21:52:04.757356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.979554+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.0019999430000000001,"stat_sum":{"num_bytes":557056,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.14","version":"38'6","reported_seq":112,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383666+0000","last_change":"2024-04-18T21:51:58.380525+0000","last_active":"2024-04-18T21:52:05.383666+0000","last_peered":"2024-04-18T21:52:05.383666+0000","last_clean":"2024-04-18T21:52:05.383666+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:05.383666+0000","last_undegraded":"2024-04-18T21:52:05.383666+0000","last_fullsized":"2024-04-18T21:52:05.383666+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.480130+0000","scrub_duration":1515,"objects_trimmed":0,"snaptrim_duration":0.212985007,"stat_sum":{"num_bytes":770048,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.650306+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:05.650306+0000","last_peered":"2024-04-18T21:52:05.650306+0000","last_clean":"2024-04-18T21:52:05.650306+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:05.650306+0000","last_undegraded":"2024-04-18T21:52:05.650306+0000","last_fullsized":"2024-04-18T21:52:05.650306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"32'2","reported_seq":105,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401570+0000","last_change":"2024-04-18T21:52:02.163455+0000","last_active":"2024-04-18T21:52:03.401570+0000","last_peered":"2024-04-18T21:52:03.401570+0000","last_clean":"2024-04-18T21:52:03.401570+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:03.401570+0000","last_undegraded":"2024-04-18T21:52:03.401570+0000","last_fullsized":"2024-04-18T21:52:03.401570+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.309515+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.0014775280000000001,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.503144+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:03.503144+0000","last_peered":"2024-04-18T21:52:03.503144+0000","last_clean":"2024-04-18T21:52:03.503144+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:03.503144+0000","last_undegraded":"2024-04-18T21:52:03.503144+0000","last_fullsized":"2024-04-18T21:52:03.503144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":99,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.502694+0000","last_change":"2024-04-18T21:51:58.176158+0000","last_active":"2024-04-18T21:52:03.502694+0000","last_peered":"2024-04-18T21:52:03.502694+0000","last_clean":"2024-04-18T21:52:03.502694+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:03.502694+0000","last_undegraded":"2024-04-18T21:52:03.502694+0000","last_fullsized":"2024-04-18T21:52:03.502694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:18.918419+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.0019225869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.650707+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:04.650707+0000","last_peered":"2024-04-18T21:52:04.650707+0000","last_clean":"2024-04-18T21:52:04.650707+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:04.650707+0000","last_undegraded":"2024-04-18T21:52:04.650707+0000","last_fullsized":"2024-04-18T21:52:04.650707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":111,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.384380+0000","last_change":"2024-04-18T21:51:58.171632+0000","last_active":"2024-04-18T21:52:05.384380+0000","last_peered":"2024-04-18T21:52:05.384380+0000","last_clean":"2024-04-18T21:52:05.384380+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:05.384380+0000","last_undegraded":"2024-04-18T21:52:05.384380+0000","last_fullsized":"2024-04-18T21:52:05.384380+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:12.138527+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.10378514599999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1585,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401143+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:03.401143+0000","last_peered":"2024-04-18T21:52:03.401143+0000","last_clean":"2024-04-18T21:52:03.401143+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:03.401143+0000","last_undegraded":"2024-04-18T21:52:03.401143+0000","last_fullsized":"2024-04-18T21:52:03.401143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"33'7","reported_seq":112,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.754684+0000","last_change":"2024-04-18T21:51:58.157867+0000","last_active":"2024-04-18T21:52:04.754684+0000","last_peered":"2024-04-18T21:52:04.754684+0000","last_clean":"2024-04-18T21:52:04.754684+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:04.754684+0000","last_undegraded":"2024-04-18T21:52:04.754684+0000","last_fullsized":"2024-04-18T21:52:04.754684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:19.858344+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.002328845,"stat_sum":{"num_bytes":835584,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.3","version":"45'58","reported_seq":186,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.450274+0000","last_change":"2024-04-18T21:52:03.915511+0000","last_active":"2024-04-18T21:52:05.450274+0000","last_peered":"2024-04-18T21:52:05.450274+0000","last_clean":"2024-04-18T21:52:05.450274+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:05.450274+0000","last_undegraded":"2024-04-18T21:52:05.450274+0000","last_fullsized":"2024-04-18T21:52:05.450274+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:20.101521+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.30771508800000003,"stat_sum":{"num_bytes":8060928,"num_objects":16,"num_object_clones":8,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.756281+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:04.756281+0000","last_peered":"2024-04-18T21:52:04.756281+0000","last_clean":"2024-04-18T21:52:04.756281+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:04.756281+0000","last_undegraded":"2024-04-18T21:52:04.756281+0000","last_fullsized":"2024-04-18T21:52:04.756281+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"42'14","reported_seq":133,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.449482+0000","last_change":"2024-04-18T21:52:01.703639+0000","last_active":"2024-04-18T21:52:05.449482+0000","last_peered":"2024-04-18T21:52:05.449482+0000","last_clean":"2024-04-18T21:52:05.449482+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:05.449482+0000","last_undegraded":"2024-04-18T21:52:05.449482+0000","last_fullsized":"2024-04-18T21:52:05.449482+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.919850+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.204535667,"stat_sum":{"num_bytes":2883584,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.599956+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:03.599956+0000","last_peered":"2024-04-18T21:52:03.599956+0000","last_clean":"2024-04-18T21:52:03.599956+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:03.599956+0000","last_undegraded":"2024-04-18T21:52:03.599956+0000","last_fullsized":"2024-04-18T21:52:03.599956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"34'8","reported_seq":115,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.400694+0000","last_change":"2024-04-18T21:51:58.075957+0000","last_active":"2024-04-18T21:52:03.400694+0000","last_peered":"2024-04-18T21:52:03.400694+0000","last_clean":"2024-04-18T21:52:03.400694+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:03.400694+0000","last_undegraded":"2024-04-18T21:52:03.400694+0000","last_fullsized":"2024-04-18T21:52:03.400694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:07.467667+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0.001275502,"stat_sum":{"num_bytes":1458176,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.463708+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:04.463708+0000","last_peered":"2024-04-18T21:52:04.463708+0000","last_clean":"2024-04-18T21:52:04.463708+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:04.463708+0000","last_undegraded":"2024-04-18T21:52:04.463708+0000","last_fullsized":"2024-04-18T21:52:04.463708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"29'2","reported_seq":113,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383155+0000","last_change":"2024-04-18T21:52:02.588926+0000","last_active":"2024-04-18T21:52:05.383155+0000","last_peered":"2024-04-18T21:52:05.383155+0000","last_clean":"2024-04-18T21:52:05.383155+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:05.383155+0000","last_undegraded":"2024-04-18T21:52:05.383155+0000","last_fullsized":"2024-04-18T21:52:05.383155+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.058469+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.10213271,"stat_sum":{"num_bytes":212992,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.757118+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:04.757118+0000","last_peered":"2024-04-18T21:52:04.757118+0000","last_clean":"2024-04-18T21:52:04.757118+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:04.757118+0000","last_undegraded":"2024-04-18T21:52:04.757118+0000","last_fullsized":"2024-04-18T21:52:04.757118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":241,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.363921+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:04.363921+0000","last_peered":"2024-04-18T21:52:04.363921+0000","last_clean":"2024-04-18T21:52:04.363921+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:04.363921+0000","last_undegraded":"2024-04-18T21:52:04.363921+0000","last_fullsized":"2024-04-18T21:52:04.363921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":104,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.750539+0000","last_change":"2024-04-18T21:52:00.379115+0000","last_active":"2024-04-18T21:52:04.750539+0000","last_peered":"2024-04-18T21:52:04.750539+0000","last_clean":"2024-04-18T21:52:04.750539+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:04.750539+0000","last_undegraded":"2024-04-18T21:52:04.750539+0000","last_fullsized":"2024-04-18T21:52:04.750539+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.029798+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.00080647200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.9","version":"29'6","reported_seq":116,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.752259+0000","last_change":"2024-04-18T21:52:02.588756+0000","last_active":"2024-04-18T21:52:04.752259+0000","last_peered":"2024-04-18T21:52:04.752259+0000","last_clean":"2024-04-18T21:52:04.752259+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:04.752259+0000","last_undegraded":"2024-04-18T21:52:04.752259+0000","last_fullsized":"2024-04-18T21:52:04.752259+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:32.361581+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.205802136,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.a","version":"41'11","reported_seq":102,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.550641+0000","last_change":"2024-04-18T21:51:58.787646+0000","last_active":"2024-04-18T21:52:05.550641+0000","last_peered":"2024-04-18T21:52:05.550641+0000","last_clean":"2024-04-18T21:52:05.550641+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:05.550641+0000","last_undegraded":"2024-04-18T21:52:05.550641+0000","last_fullsized":"2024-04-18T21:52:05.550641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:45.279750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51759911599999997,"stat_sum":{"num_bytes":1261568,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.b","version":"49'32","reported_seq":122,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.700061+0000","last_change":"2024-04-18T21:51:58.887472+0000","last_active":"2024-04-18T21:52:05.700061+0000","last_peered":"2024-04-18T21:52:05.700061+0000","last_clean":"2024-04-18T21:52:05.700061+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:05.700061+0000","last_undegraded":"2024-04-18T21:52:05.700061+0000","last_fullsized":"2024-04-18T21:52:05.700061+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:50.009470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.72128645800000002,"stat_sum":{"num_bytes":5472256,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.c","version":"41'23","reported_seq":117,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.756712+0000","last_change":"2024-04-18T21:51:58.468330+0000","last_active":"2024-04-18T21:52:04.756712+0000","last_peered":"2024-04-18T21:52:04.756712+0000","last_clean":"2024-04-18T21:52:04.756712+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:04.756712+0000","last_undegraded":"2024-04-18T21:52:04.756712+0000","last_fullsized":"2024-04-18T21:52:04.756712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.442380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21306905500000001,"stat_sum":{"num_bytes":3915776,"num_objects":9,"num_object_clones":4,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":22,"num_read_kb":2008,"num_write":36,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":84,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.364507+0000","last_change":"2024-04-18T21:51:58.155757+0000","last_active":"2024-04-18T21:52:04.364507+0000","last_peered":"2024-04-18T21:52:04.364507+0000","last_clean":"2024-04-18T21:52:04.364507+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:04.364507+0000","last_undegraded":"2024-04-18T21:52:04.364507+0000","last_fullsized":"2024-04-18T21:52:04.364507+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:04.424721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00143743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.e","version":"43'24","reported_seq":118,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.364670+0000","last_change":"2024-04-18T21:51:58.155377+0000","last_active":"2024-04-18T21:52:04.364670+0000","last_peered":"2024-04-18T21:52:04.364670+0000","last_clean":"2024-04-18T21:52:04.364670+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:04.364670+0000","last_undegraded":"2024-04-18T21:52:04.364670+0000","last_fullsized":"2024-04-18T21:52:04.364670+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.054741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002880685,"stat_sum":{"num_bytes":2785280,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":4039,"num_write":17,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.f","version":"41'7","reported_seq":98,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.365041+0000","last_change":"2024-04-18T21:51:58.155833+0000","last_active":"2024-04-18T21:52:04.365041+0000","last_peered":"2024-04-18T21:52:04.365041+0000","last_clean":"2024-04-18T21:52:04.365041+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:04.365041+0000","last_undegraded":"2024-04-18T21:52:04.365041+0000","last_fullsized":"2024-04-18T21:52:04.365041+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:08.333087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014725999999999999,"stat_sum":{"num_bytes":491520,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":2884,"num_write":6,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.10","version":"41'16","reported_seq":89,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.550065+0000","last_change":"2024-04-18T21:52:03.761892+0000","last_active":"2024-04-18T21:52:05.550065+0000","last_peered":"2024-04-18T21:52:05.550065+0000","last_clean":"2024-04-18T21:52:05.550065+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:05.550065+0000","last_undegraded":"2024-04-18T21:52:05.550065+0000","last_fullsized":"2024-04-18T21:52:05.550065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:16.187800+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0.103603797,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.11","version":"47'16","reported_seq":47,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-18T21:52:05.055442+0000","last_change":"2024-04-18T21:52:04.346146+0000","last_active":"2024-04-18T21:52:05.055442+0000","last_peered":"2024-04-18T21:52:05.055442+0000","last_clean":"2024-04-18T21:51:50.243799+0000","last_became_active":"2024-04-18T21:52:02.703977+0000","last_became_peered":"2024-04-18T21:52:02.703977+0000","last_unstale":"2024-04-18T21:52:05.055442+0000","last_undegraded":"2024-04-18T21:52:05.055442+0000","last_fullsized":"2024-04-18T21:51:50.243799+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3235840,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":3320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,2147483647,1,5,2,2147483647],"avail_no_missing":["3(0)","1(4)","2(5)","2(6)","3(2)","4(1)","5(4)","5(5)","6(2)","6(6)","7(7)","8(7)","9(3)","10(3)","11(0)"],"object_location_counts":[{"shards":"1(4),2(6),3(0),4(1),5(5),6(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.12","version":"33'5","reported_seq":114,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383126+0000","last_change":"2024-04-18T21:52:01.496556+0000","last_active":"2024-04-18T21:52:05.383126+0000","last_peered":"2024-04-18T21:52:05.383126+0000","last_clean":"2024-04-18T21:52:05.383126+0000","last_became_active":"2024-04-18T21:51:48.492285+0000","last_became_peered":"2024-04-18T21:51:48.492285+0000","last_unstale":"2024-04-18T21:52:05.383126+0000","last_undegraded":"2024-04-18T21:52:05.383126+0000","last_fullsized":"2024-04-18T21:52:05.383126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'5","last_scrub_stamp":"2024-04-18T21:52:01.496294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.496294+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.628661+0000","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0.10167762499999999,"stat_sum":{"num_bytes":819200,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.13","version":"45'22","reported_seq":134,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.449983+0000","last_change":"2024-04-18T21:52:02.183412+0000","last_active":"2024-04-18T21:52:05.449983+0000","last_peered":"2024-04-18T21:52:05.449983+0000","last_clean":"2024-04-18T21:52:05.449983+0000","last_became_active":"2024-04-18T21:51:48.486639+0000","last_became_peered":"2024-04-18T21:51:48.486639+0000","last_unstale":"2024-04-18T21:52:05.449983+0000","last_undegraded":"2024-04-18T21:52:05.449983+0000","last_fullsized":"2024-04-18T21:52:05.449983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21032728000000001,"stat_sum":{"num_bytes":3137536,"num_objects":9,"num_object_clones":3,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":20,"num_read_kb":2981,"num_write":15,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":44793856,"num_objects":88,"num_object_clones":36,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":6,"num_read":104,"num_read_kb":14039,"num_write":118,"num_write_kb":22080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87359488,"data_stored":106635264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378635,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37648,"kb_used_data":6468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334192,"statfs":{"total":96636764160,"available":96598212608,"internally_reserved":0,"allocated":6623232,"data_stored":8016391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378635,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40332,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331508,"statfs":{"total":96636764160,"available":96595464192,"internally_reserved":0,"allocated":9359360,"data_stored":10891585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378636,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39940,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331900,"statfs":{"total":96636764160,"available":96595865600,"internally_reserved":0,"allocated":8970240,"data_stored":10772999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378635,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40604,"kb_used_data":9412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331236,"statfs":{"total":96636764160,"available":96595185664,"internally_reserved":0,"allocated":9637888,"data_stored":11514177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378635,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39132,"kb_used_data":7944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332708,"statfs":{"total":96636764160,"available":96596692992,"internally_reserved":0,"allocated":8134656,"data_stored":9728099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,16,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411338,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":4972544,"data_stored":5701731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32600,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":5701632,"data_stored":6546273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411338,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40220,"kb_used_data":9040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331620,"statfs":{"total":96636764160,"available":96595578880,"internally_reserved":0,"allocated":9256960,"data_stored":10944833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411339,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37276,"kb_used_data":10244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334564,"statfs":{"total":96636764160,"available":96598593536,"internally_reserved":0,"allocated":10489856,"data_stored":12472641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411337,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34452,"kb_used_data":7424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337388,"statfs":{"total":96636764160,"available":96601485312,"internally_reserved":0,"allocated":7602176,"data_stored":8884867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444042,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34140,"kb_used_data":7112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337700,"statfs":{"total":96636764160,"available":96601804800,"internally_reserved":0,"allocated":7282688,"data_stored":8233059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444042,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":6033408,"data_stored":6774883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6553600,"data_stored":8073216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9441280,"data_stored":11612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9961472,"data_stored":12181504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7618560,"data_stored":9445376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8429568,"data_stored":10473472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8728576,"data_stored":10653696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":10989568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4640768,"data_stored":5726208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5517312,"data_stored":6492160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4456448,"data_stored":5419008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6082560,"data_stored":7618560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6766592,"data_stored":7950336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:08.157 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-18T21:52:08.164 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:08.164 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:08.172 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:08.173 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:08.179 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:08.204 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:08.204 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:08.224 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:08.222+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:08.302 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:08.299+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:08.316 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:08.317 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:08.337 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:08.337 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:08.346 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:08.347 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:08.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:08.403 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:08.399+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:08.410 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:08.411 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:08.438 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:08.439 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.467 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.457+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.468 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.469 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.470 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.471 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.477 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.477 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.477 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.477 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.478 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.479 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.480 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.481 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.482 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.461+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.483 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.484 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.485 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.486 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.487 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.488 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.489 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.490 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.491 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.465+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.491 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:08.491 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:08.491 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:08.491 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:08.491 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:08.502 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:08.503 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:08.499+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:08.531 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:08.531 INFO:teuthology.orchestra.run.smithi097.stdout:{"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":50,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":12,"num_up_osds":12,"osd_up_since":1713477089,"num_in_osds":12,"osd_in_since":1713477081,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":34},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":35,"num_pools":3,"num_objects":90,"data_bytes":45384224,"bytes_used":451735552,"bytes_avail":1159189434368,"bytes_total":1159641169920},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-18T21:51:17.632974+0000","services":{}},"progress_events":{}} 2024-04-18T21:52:08.548 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-18T21:52:08.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:08.573 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.574 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.575 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.576 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.577 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.578 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.578 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.578 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.578 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.578 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.578 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.578 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.578 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.565+0000 7f9d7974e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.604 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:08.601+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:08.621 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:108 3.6s0 3:64491ca4:::smithi09731216-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.680309+0000", 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.941998184999999, 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29480844099999998, 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.622 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 108 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680309+0000", 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680309+0000", 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680311+0000", 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6400000000000001e-06 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680323+0000", 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.146e-05 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680355+0000", 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2149e-05 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680378+0000", 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2707000000000001e-05 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.787116+0000", 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10673869900000001 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.787340+0000", 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00022391800000000001 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.975082+0000", 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.187741879 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.975117+0000", 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4989e-05 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.626 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:106 3.6s0 3:64491ca4:::smithi09731216-36:head [append 0~589824,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.533558364000001, 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.39237912, 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 106 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088750+0000", 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3859999999999999e-06 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089602+0000", 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000852537 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089629+0000", 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6066000000000001e-05 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089648+0000", 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9449999999999998e-05 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174464+0000", 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816027000000002 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174469+0000", 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5970000000000002e-06 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481023+0000", 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3065548339999999 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481128+0000", 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010422299999999999 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:107 3.6s0 3:64491ca4:::smithi09731216-36:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.532629401000001, 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.301432395, 2024-04-18T21:52:08.631 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 107 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089679+0000", 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.686e-06 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089697+0000", 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7717e-05 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089714+0000", 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7433000000000001e-05 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089730+0000", 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5953000000000001e-05 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174547+0000", 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816707000000005 2024-04-18T21:52:08.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174550+0000", 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9100000000000001e-06 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.983958+0000", 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8094078099999999 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984087+0000", 2024-04-18T21:52:08.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012884800000000001 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984411+0000", 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032455500000000003 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986825+0000", 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024135609999999998 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391001+0000", 2024-04-18T21:52:08.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40417611399999998 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391110+0000", 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109101 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:108 3.6s0 3:64491ca4:::smithi09731216-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.532555756000001, 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4199299170000002, 2024-04-18T21:52:08.637 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 108 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089753+0000", 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.46e-06 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089759+0000", 2024-04-18T21:52:08.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5969999999999997e-06 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089782+0000", 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2546999999999999e-05 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089799+0000", 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6742999999999999e-05 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174653+0000", 2024-04-18T21:52:08.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084854180000000001 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.886e-06 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391631+0000", 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2169758150000001 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391728+0000", 2024-04-18T21:52:08.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6861000000000007e-05 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509681+0000", 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.117952828 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:109 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~606208,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:08.642 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.532482203000001, 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4179864210000002, 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:08.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089826+0000", 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.147e-06 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090694+0000", 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00086783200000000002 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090726+0000", 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2379999999999998e-05 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090746+0000", 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9466999999999998e-05 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174599+0000", 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083853091000000005 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174600+0000", 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.097e-06 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985378+0000", 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.810778236 2024-04-18T21:52:08.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985469+0000", 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0805999999999998e-05 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986312+0000", 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084315300000000002 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089266+0000", 2024-04-18T21:52:08.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10295370099999999 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507611+0000", 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41834516199999999 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507811+0000", 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000200349 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:110 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.5315388, 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.429779607, 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.649 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 110 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090769+0000", 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090782+0000", 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2547e-05 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090797+0000", 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5109000000000001e-05 2024-04-18T21:52:08.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090807+0000", 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8670000000000006e-06 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083849218000000003 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174657+0000", 2024-04-18T21:52:08.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8700000000000004e-07 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087432+0000", 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9127752569999998 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087519+0000", 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6668000000000004e-05 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088034+0000", 2024-04-18T21:52:08.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00051538599999999995 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089552+0000", 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001518295 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520465+0000", 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.43091233000000001 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520548+0000", 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2947000000000004e-05 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:111 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.531489060999998, 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5330603059999999, 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.655 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 111 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090819+0000", 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6000000000000003e-07 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090822+0000", 2024-04-18T21:52:08.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.066e-06 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090834+0000", 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2486999999999999e-05 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090844+0000", 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9899999999999992e-06 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275049+0000", 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.184205065 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275055+0000", 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4389999999999999e-06 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088362+0000", 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8133068830000001 2024-04-18T21:52:08.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088621+0000", 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025974499999999997 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520968+0000", 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.432346972 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521091+0000", 2024-04-18T21:52:08.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122263 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.623878+0000", 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102787636 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:127 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~720896,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.808014075999999, 2024-04-18T21:52:08.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50504046999999996, 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 127 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:08.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814297+0000", 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3120000000000004e-06 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815282+0000", 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0009845469999999999 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815306+0000", 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4488999999999999e-05 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815321+0000", 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4365999999999999e-05 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901001+0000", 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085680191000000003 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901126+0000", 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125506 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902346+0000", 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012201989999999999 2024-04-18T21:52:08.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904676+0000", 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023299459999999998 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319226+0000", 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.414549324 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319333+0000", 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010759 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:128 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.806959320000001, 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70473296200000002, 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:08.667 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 128 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815349+0000", 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.003e-06 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815353+0000", 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5700000000000003e-06 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815362+0000", 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4540000000000007e-06 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815369+0000", 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4059999999999999e-06 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903371+0000", 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088001653999999999 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903511+0000", 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000139889 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904181+0000", 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067014400000000001 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.905274+0000", 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010929430000000001 2024-04-18T21:52:08.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519876+0000", 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61460184500000004 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520081+0000", 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020505400000000001 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.1s0,GRANT,e:32,from:5(4),reservation_nonce:5)", 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.72077307, 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61973115499999998, 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901535+0000", 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.206e-06 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901541+0000", 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.331e-06 2024-04-18T21:52:08.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901559+0000", 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7977000000000001e-05 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.001690+0000", 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100130964 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521089+0000", 2024-04-18T21:52:08.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51939875099999999 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521142+0000", 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3324e-05 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521265+0000", 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122602 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.676 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:130 3.1s0 3:87c089d9:::smithi09731216-44:head [append 0~737280,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.711300801, 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61075046099999997, 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 130 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911008+0000", 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.356e-06 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911859+0000", 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085008399999999997 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911877+0000", 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8147000000000001e-05 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911886+0000", 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012129+0000", 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100243058 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012391+0000", 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026173100000000002 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.014631+0000", 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002240344 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:08.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118257+0000", 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103625859 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521564+0000", 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40330724200000001 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521757+0000", 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019236000000000001 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.681 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:131 3.1s0 3:87c089d9:::smithi09731216-44:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.710402492, 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70851564899999997, 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:08.682 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 131 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911905+0000", 2024-04-18T21:52:08.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9800000000000002e-07 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911910+0000", 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4499999999999997e-06 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911919+0000", 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911927+0000", 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5700000000000004e-06 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017049+0000", 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105122775 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017158+0000", 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010845600000000001 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017640+0000", 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00048220899999999999 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118261+0000", 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100621063 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620272+0000", 2024-04-18T21:52:08.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50201088400000005 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620420+0000", 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000148064 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 32/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:08.687 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.602537837, 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29966542299999999, 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019772+0000", 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2869999999999999e-06 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019780+0000", 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8990000000000001e-06 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019783+0000", 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8460000000000001e-06 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319409+0000", 2024-04-18T21:52:08.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29962544499999999 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319435+0000", 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5605999999999998e-05 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:08.691 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.600279045000001, 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.297022972, 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022030+0000", 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7940000000000001e-06 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022039+0000", 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6339999999999996e-06 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022063+0000", 2024-04-18T21:52:08.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4026e-05 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022074+0000", 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0705e-05 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319012+0000", 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29693840300000002 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319051+0000", 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.841e-05 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 33/31 ECSubWriteReply(tid=15, last_complete=32'3, committed=1, applied=1))", 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.532850429, 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30109323399999999, 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089458+0000", 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5859999999999999e-06 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089467+0000", 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0420000000000008e-06 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089492+0000", 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5211999999999998e-05 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.189619+0000", 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100126885 2024-04-18T21:52:08.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.390510+0000", 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20089099299999999 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.390550+0000", 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9515999999999999e-05 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.515418050000001, 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31260053300000001, 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106890+0000", 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8599999999999996e-07 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106896+0000", 2024-04-18T21:52:08.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9139999999999996e-06 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106911+0000", 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4800999999999999e-05 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106920+0000", 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9379999999999998e-06 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419470+0000", 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.312550204 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419489+0000", 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.969e-05 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:08.702 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.417493276999998, 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.316195273, 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1699999999999998e-07 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204818+0000", 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5659999999999998e-06 2024-04-18T21:52:08.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204828+0000", 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0478e-05 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204836+0000", 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.413e-06 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519654+0000", 2024-04-18T21:52:08.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.314818722 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521009+0000", 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001354577 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=13, last_complete=32'5, committed=1, applied=1))", 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:08.706 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.417459592, 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31487986099999998, 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:08.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204848+0000", 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8299999999999998e-07 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204850+0000", 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6429999999999999e-06 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204859+0000", 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0550000000000005e-06 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204866+0000", 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5320000000000003e-06 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519669+0000", 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31480259199999999 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519727+0000", 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8656000000000002e-05 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.417426734999999, 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31508690700000003, 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.710 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204881+0000", 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3000000000000002e-07 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204884+0000", 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.976e-06 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204900+0000", 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6418000000000001e-05 2024-04-18T21:52:08.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204908+0000", 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3080000000000008e-06 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519923+0000", 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31501511199999999 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519967+0000", 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3763000000000001e-05 2024-04-18T21:52:08.713 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.414677558000001, 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31221361399999997, 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:08.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207631+0000", 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.42e-06 2024-04-18T21:52:08.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207640+0000", 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7350000000000002e-06 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207668+0000", 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8881999999999999e-05 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207686+0000", 2024-04-18T21:52:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7136999999999999e-05 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519798+0000", 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31211196099999999 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519843+0000", 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5479000000000003e-05 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.717 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:08.718 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:08.718 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:08.719 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:08.717+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:08.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:08.797 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:08.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:08.820 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:08.821+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:08.839 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:08.839 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:08.839 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:08.840 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:08.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:08.890 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:08.891 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:08.909 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-04-18T21:52:08.356528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":45384224,"num_objects":90,"num_object_clones":36,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":6,"num_read":230,"num_read_kb":14148,"num_write":299,"num_write_kb":25018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":231,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":441148,"kb_used_data":91860,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1132020932,"statfs":{"total":1159641169920,"available":1159189434368,"internally_reserved":0,"allocated":94064640,"data_stored":110481538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":357414205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1,16,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5611520,"num_objects":9,"num_object_clones":5,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":-69,"num_read_kb":-9286,"num_write":-104,"num_write_kb":-12664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-376832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.995508"},"pg_stats":[{"pgid":"3.19","version":"41'9","reported_seq":121,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.853140+0000","last_change":"2024-04-18T21:52:01.767047+0000","last_active":"2024-04-18T21:52:04.853140+0000","last_peered":"2024-04-18T21:52:04.853140+0000","last_clean":"2024-04-18T21:52:04.853140+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:04.853140+0000","last_undegraded":"2024-04-18T21:52:04.853140+0000","last_fullsized":"2024-04-18T21:52:04.853140+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:12.682126+0000","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0.102052106,"stat_sum":{"num_bytes":262144,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.18","version":"42'16","reported_seq":123,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.750839+0000","last_change":"2024-04-18T21:52:00.261138+0000","last_active":"2024-04-18T21:52:04.750839+0000","last_peered":"2024-04-18T21:52:04.750839+0000","last_clean":"2024-04-18T21:52:04.750839+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:04.750839+0000","last_undegraded":"2024-04-18T21:52:04.750839+0000","last_fullsized":"2024-04-18T21:52:04.750839+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.847030+0000","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0.0011527200000000001,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.17","version":"39'6","reported_seq":108,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.502201+0000","last_change":"2024-04-18T21:51:59.786767+0000","last_active":"2024-04-18T21:52:03.502201+0000","last_peered":"2024-04-18T21:52:03.502201+0000","last_clean":"2024-04-18T21:52:03.502201+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:03.502201+0000","last_undegraded":"2024-04-18T21:52:03.502201+0000","last_fullsized":"2024-04-18T21:52:03.502201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.350212+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.10179099799999999,"stat_sum":{"num_bytes":671744,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.16","version":"32'3","reported_seq":109,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401291+0000","last_change":"2024-04-18T21:52:01.391849+0000","last_active":"2024-04-18T21:52:03.401291+0000","last_peered":"2024-04-18T21:52:03.401291+0000","last_clean":"2024-04-18T21:52:03.401291+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:03.401291+0000","last_undegraded":"2024-04-18T21:52:03.401291+0000","last_fullsized":"2024-04-18T21:52:03.401291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.764499+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.0012794480000000001,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.15","version":"39'7","reported_seq":113,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.757356+0000","last_change":"2024-04-18T21:51:58.257029+0000","last_active":"2024-04-18T21:52:04.757356+0000","last_peered":"2024-04-18T21:52:04.757356+0000","last_clean":"2024-04-18T21:52:04.757356+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:04.757356+0000","last_undegraded":"2024-04-18T21:52:04.757356+0000","last_fullsized":"2024-04-18T21:52:04.757356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.979554+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.0019999430000000001,"stat_sum":{"num_bytes":557056,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.14","version":"38'6","reported_seq":112,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383666+0000","last_change":"2024-04-18T21:51:58.380525+0000","last_active":"2024-04-18T21:52:05.383666+0000","last_peered":"2024-04-18T21:52:05.383666+0000","last_clean":"2024-04-18T21:52:05.383666+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:05.383666+0000","last_undegraded":"2024-04-18T21:52:05.383666+0000","last_fullsized":"2024-04-18T21:52:05.383666+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.480130+0000","scrub_duration":1515,"objects_trimmed":0,"snaptrim_duration":0.212985007,"stat_sum":{"num_bytes":770048,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.650306+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:05.650306+0000","last_peered":"2024-04-18T21:52:05.650306+0000","last_clean":"2024-04-18T21:52:05.650306+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:05.650306+0000","last_undegraded":"2024-04-18T21:52:05.650306+0000","last_fullsized":"2024-04-18T21:52:05.650306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"32'2","reported_seq":105,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401570+0000","last_change":"2024-04-18T21:52:02.163455+0000","last_active":"2024-04-18T21:52:03.401570+0000","last_peered":"2024-04-18T21:52:03.401570+0000","last_clean":"2024-04-18T21:52:03.401570+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:03.401570+0000","last_undegraded":"2024-04-18T21:52:03.401570+0000","last_fullsized":"2024-04-18T21:52:03.401570+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.309515+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.0014775280000000001,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.503144+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:03.503144+0000","last_peered":"2024-04-18T21:52:03.503144+0000","last_clean":"2024-04-18T21:52:03.503144+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:03.503144+0000","last_undegraded":"2024-04-18T21:52:03.503144+0000","last_fullsized":"2024-04-18T21:52:03.503144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":99,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.502694+0000","last_change":"2024-04-18T21:51:58.176158+0000","last_active":"2024-04-18T21:52:03.502694+0000","last_peered":"2024-04-18T21:52:03.502694+0000","last_clean":"2024-04-18T21:52:03.502694+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:03.502694+0000","last_undegraded":"2024-04-18T21:52:03.502694+0000","last_fullsized":"2024-04-18T21:52:03.502694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:18.918419+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.0019225869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.650707+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:04.650707+0000","last_peered":"2024-04-18T21:52:04.650707+0000","last_clean":"2024-04-18T21:52:04.650707+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:04.650707+0000","last_undegraded":"2024-04-18T21:52:04.650707+0000","last_fullsized":"2024-04-18T21:52:04.650707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":111,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.384380+0000","last_change":"2024-04-18T21:51:58.171632+0000","last_active":"2024-04-18T21:52:05.384380+0000","last_peered":"2024-04-18T21:52:05.384380+0000","last_clean":"2024-04-18T21:52:05.384380+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:05.384380+0000","last_undegraded":"2024-04-18T21:52:05.384380+0000","last_fullsized":"2024-04-18T21:52:05.384380+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:12.138527+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.10378514599999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1585,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.401143+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:03.401143+0000","last_peered":"2024-04-18T21:52:03.401143+0000","last_clean":"2024-04-18T21:52:03.401143+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:03.401143+0000","last_undegraded":"2024-04-18T21:52:03.401143+0000","last_fullsized":"2024-04-18T21:52:03.401143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"33'7","reported_seq":112,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.754684+0000","last_change":"2024-04-18T21:51:58.157867+0000","last_active":"2024-04-18T21:52:04.754684+0000","last_peered":"2024-04-18T21:52:04.754684+0000","last_clean":"2024-04-18T21:52:04.754684+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:04.754684+0000","last_undegraded":"2024-04-18T21:52:04.754684+0000","last_fullsized":"2024-04-18T21:52:04.754684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:19.858344+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.002328845,"stat_sum":{"num_bytes":835584,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.3","version":"45'58","reported_seq":186,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.450274+0000","last_change":"2024-04-18T21:52:03.915511+0000","last_active":"2024-04-18T21:52:05.450274+0000","last_peered":"2024-04-18T21:52:05.450274+0000","last_clean":"2024-04-18T21:52:05.450274+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:05.450274+0000","last_undegraded":"2024-04-18T21:52:05.450274+0000","last_fullsized":"2024-04-18T21:52:05.450274+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:20.101521+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.30771508800000003,"stat_sum":{"num_bytes":8060928,"num_objects":16,"num_object_clones":8,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.756281+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:04.756281+0000","last_peered":"2024-04-18T21:52:04.756281+0000","last_clean":"2024-04-18T21:52:04.756281+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:04.756281+0000","last_undegraded":"2024-04-18T21:52:04.756281+0000","last_fullsized":"2024-04-18T21:52:04.756281+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"42'14","reported_seq":133,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.449482+0000","last_change":"2024-04-18T21:52:01.703639+0000","last_active":"2024-04-18T21:52:05.449482+0000","last_peered":"2024-04-18T21:52:05.449482+0000","last_clean":"2024-04-18T21:52:05.449482+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:05.449482+0000","last_undegraded":"2024-04-18T21:52:05.449482+0000","last_fullsized":"2024-04-18T21:52:05.449482+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.919850+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.204535667,"stat_sum":{"num_bytes":2883584,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.599956+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:03.599956+0000","last_peered":"2024-04-18T21:52:03.599956+0000","last_clean":"2024-04-18T21:52:03.599956+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:03.599956+0000","last_undegraded":"2024-04-18T21:52:03.599956+0000","last_fullsized":"2024-04-18T21:52:03.599956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"34'8","reported_seq":115,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-04-18T21:52:03.400694+0000","last_change":"2024-04-18T21:51:58.075957+0000","last_active":"2024-04-18T21:52:03.400694+0000","last_peered":"2024-04-18T21:52:03.400694+0000","last_clean":"2024-04-18T21:52:03.400694+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:03.400694+0000","last_undegraded":"2024-04-18T21:52:03.400694+0000","last_fullsized":"2024-04-18T21:52:03.400694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:07.467667+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0.001275502,"stat_sum":{"num_bytes":1458176,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.463708+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:04.463708+0000","last_peered":"2024-04-18T21:52:04.463708+0000","last_clean":"2024-04-18T21:52:04.463708+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:04.463708+0000","last_undegraded":"2024-04-18T21:52:04.463708+0000","last_fullsized":"2024-04-18T21:52:04.463708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"29'2","reported_seq":113,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383155+0000","last_change":"2024-04-18T21:52:02.588926+0000","last_active":"2024-04-18T21:52:05.383155+0000","last_peered":"2024-04-18T21:52:05.383155+0000","last_clean":"2024-04-18T21:52:05.383155+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:05.383155+0000","last_undegraded":"2024-04-18T21:52:05.383155+0000","last_fullsized":"2024-04-18T21:52:05.383155+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.058469+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.10213271,"stat_sum":{"num_bytes":212992,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.757118+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:04.757118+0000","last_peered":"2024-04-18T21:52:04.757118+0000","last_clean":"2024-04-18T21:52:04.757118+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:04.757118+0000","last_undegraded":"2024-04-18T21:52:04.757118+0000","last_fullsized":"2024-04-18T21:52:04.757118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":241,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.363921+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:04.363921+0000","last_peered":"2024-04-18T21:52:04.363921+0000","last_clean":"2024-04-18T21:52:04.363921+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:04.363921+0000","last_undegraded":"2024-04-18T21:52:04.363921+0000","last_fullsized":"2024-04-18T21:52:04.363921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":104,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.750539+0000","last_change":"2024-04-18T21:52:00.379115+0000","last_active":"2024-04-18T21:52:04.750539+0000","last_peered":"2024-04-18T21:52:04.750539+0000","last_clean":"2024-04-18T21:52:04.750539+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:04.750539+0000","last_undegraded":"2024-04-18T21:52:04.750539+0000","last_fullsized":"2024-04-18T21:52:04.750539+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.029798+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.00080647200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.9","version":"29'6","reported_seq":116,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.752259+0000","last_change":"2024-04-18T21:52:02.588756+0000","last_active":"2024-04-18T21:52:04.752259+0000","last_peered":"2024-04-18T21:52:04.752259+0000","last_clean":"2024-04-18T21:52:04.752259+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:04.752259+0000","last_undegraded":"2024-04-18T21:52:04.752259+0000","last_fullsized":"2024-04-18T21:52:04.752259+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:32.361581+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.205802136,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.a","version":"41'11","reported_seq":102,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.550641+0000","last_change":"2024-04-18T21:51:58.787646+0000","last_active":"2024-04-18T21:52:05.550641+0000","last_peered":"2024-04-18T21:52:05.550641+0000","last_clean":"2024-04-18T21:52:05.550641+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:05.550641+0000","last_undegraded":"2024-04-18T21:52:05.550641+0000","last_fullsized":"2024-04-18T21:52:05.550641+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:45.279750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51759911599999997,"stat_sum":{"num_bytes":1261568,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.b","version":"49'32","reported_seq":122,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.700061+0000","last_change":"2024-04-18T21:51:58.887472+0000","last_active":"2024-04-18T21:52:05.700061+0000","last_peered":"2024-04-18T21:52:05.700061+0000","last_clean":"2024-04-18T21:52:05.700061+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:05.700061+0000","last_undegraded":"2024-04-18T21:52:05.700061+0000","last_fullsized":"2024-04-18T21:52:05.700061+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:50.009470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.72128645800000002,"stat_sum":{"num_bytes":5472256,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.c","version":"41'23","reported_seq":117,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.756712+0000","last_change":"2024-04-18T21:51:58.468330+0000","last_active":"2024-04-18T21:52:04.756712+0000","last_peered":"2024-04-18T21:52:04.756712+0000","last_clean":"2024-04-18T21:52:04.756712+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:04.756712+0000","last_undegraded":"2024-04-18T21:52:04.756712+0000","last_fullsized":"2024-04-18T21:52:04.756712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.442380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21306905500000001,"stat_sum":{"num_bytes":3915776,"num_objects":9,"num_object_clones":4,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":22,"num_read_kb":2008,"num_write":36,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":84,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.364507+0000","last_change":"2024-04-18T21:51:58.155757+0000","last_active":"2024-04-18T21:52:04.364507+0000","last_peered":"2024-04-18T21:52:04.364507+0000","last_clean":"2024-04-18T21:52:04.364507+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:04.364507+0000","last_undegraded":"2024-04-18T21:52:04.364507+0000","last_fullsized":"2024-04-18T21:52:04.364507+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:04.424721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00143743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.e","version":"43'24","reported_seq":118,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.364670+0000","last_change":"2024-04-18T21:51:58.155377+0000","last_active":"2024-04-18T21:52:04.364670+0000","last_peered":"2024-04-18T21:52:04.364670+0000","last_clean":"2024-04-18T21:52:04.364670+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:04.364670+0000","last_undegraded":"2024-04-18T21:52:04.364670+0000","last_fullsized":"2024-04-18T21:52:04.364670+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.054741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002880685,"stat_sum":{"num_bytes":2785280,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":4039,"num_write":17,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.f","version":"41'7","reported_seq":98,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-18T21:52:04.365041+0000","last_change":"2024-04-18T21:51:58.155833+0000","last_active":"2024-04-18T21:52:04.365041+0000","last_peered":"2024-04-18T21:52:04.365041+0000","last_clean":"2024-04-18T21:52:04.365041+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:04.365041+0000","last_undegraded":"2024-04-18T21:52:04.365041+0000","last_fullsized":"2024-04-18T21:52:04.365041+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:08.333087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014725999999999999,"stat_sum":{"num_bytes":491520,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":2884,"num_write":6,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.10","version":"41'16","reported_seq":89,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.550065+0000","last_change":"2024-04-18T21:52:03.761892+0000","last_active":"2024-04-18T21:52:05.550065+0000","last_peered":"2024-04-18T21:52:05.550065+0000","last_clean":"2024-04-18T21:52:05.550065+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:05.550065+0000","last_undegraded":"2024-04-18T21:52:05.550065+0000","last_fullsized":"2024-04-18T21:52:05.550065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:16.187800+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0.103603797,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.11","version":"47'16","reported_seq":47,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-18T21:52:05.055442+0000","last_change":"2024-04-18T21:52:04.346146+0000","last_active":"2024-04-18T21:52:05.055442+0000","last_peered":"2024-04-18T21:52:05.055442+0000","last_clean":"2024-04-18T21:51:50.243799+0000","last_became_active":"2024-04-18T21:52:02.703977+0000","last_became_peered":"2024-04-18T21:52:02.703977+0000","last_unstale":"2024-04-18T21:52:05.055442+0000","last_undegraded":"2024-04-18T21:52:05.055442+0000","last_fullsized":"2024-04-18T21:51:50.243799+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3235840,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":3320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,2147483647,1,5,2,2147483647],"avail_no_missing":["3(0)","1(4)","2(5)","2(6)","3(2)","4(1)","5(4)","5(5)","6(2)","6(6)","7(7)","8(7)","9(3)","10(3)","11(0)"],"object_location_counts":[{"shards":"1(4),2(6),3(0),4(1),5(5),6(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.12","version":"33'5","reported_seq":114,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.383126+0000","last_change":"2024-04-18T21:52:01.496556+0000","last_active":"2024-04-18T21:52:05.383126+0000","last_peered":"2024-04-18T21:52:05.383126+0000","last_clean":"2024-04-18T21:52:05.383126+0000","last_became_active":"2024-04-18T21:51:48.492285+0000","last_became_peered":"2024-04-18T21:51:48.492285+0000","last_unstale":"2024-04-18T21:52:05.383126+0000","last_undegraded":"2024-04-18T21:52:05.383126+0000","last_fullsized":"2024-04-18T21:52:05.383126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'5","last_scrub_stamp":"2024-04-18T21:52:01.496294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.496294+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.628661+0000","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0.10167762499999999,"stat_sum":{"num_bytes":819200,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]},{"pgid":"3.13","version":"45'22","reported_seq":134,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-18T21:52:05.449983+0000","last_change":"2024-04-18T21:52:02.183412+0000","last_active":"2024-04-18T21:52:05.449983+0000","last_peered":"2024-04-18T21:52:05.449983+0000","last_clean":"2024-04-18T21:52:05.449983+0000","last_became_active":"2024-04-18T21:51:48.486639+0000","last_became_peered":"2024-04-18T21:51:48.486639+0000","last_unstale":"2024-04-18T21:52:05.449983+0000","last_undegraded":"2024-04-18T21:52:05.449983+0000","last_fullsized":"2024-04-18T21:52:05.449983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21032728000000001,"stat_sum":{"num_bytes":3137536,"num_objects":9,"num_object_clones":3,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":20,"num_read_kb":2981,"num_write":15,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":44793856,"num_objects":88,"num_object_clones":36,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":6,"num_read":104,"num_read_kb":14039,"num_write":118,"num_write_kb":22080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87359488,"data_stored":106635264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378635,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37648,"kb_used_data":6468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334192,"statfs":{"total":96636764160,"available":96598212608,"internally_reserved":0,"allocated":6623232,"data_stored":8016391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378635,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40332,"kb_used_data":9140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331508,"statfs":{"total":96636764160,"available":96595464192,"internally_reserved":0,"allocated":9359360,"data_stored":10891585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378636,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39940,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331900,"statfs":{"total":96636764160,"available":96595865600,"internally_reserved":0,"allocated":8970240,"data_stored":10772999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378635,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40604,"kb_used_data":9412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331236,"statfs":{"total":96636764160,"available":96595185664,"internally_reserved":0,"allocated":9637888,"data_stored":11514177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378635,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39132,"kb_used_data":7944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332708,"statfs":{"total":96636764160,"available":96596692992,"internally_reserved":0,"allocated":8134656,"data_stored":9728099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,16,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411338,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":4972544,"data_stored":5701731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32600,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":5701632,"data_stored":6546273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411338,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40220,"kb_used_data":9040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331620,"statfs":{"total":96636764160,"available":96595578880,"internally_reserved":0,"allocated":9256960,"data_stored":10944833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411339,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37276,"kb_used_data":10244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334564,"statfs":{"total":96636764160,"available":96598593536,"internally_reserved":0,"allocated":10489856,"data_stored":12472641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411337,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34452,"kb_used_data":7424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337388,"statfs":{"total":96636764160,"available":96601485312,"internally_reserved":0,"allocated":7602176,"data_stored":8884867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444042,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34140,"kb_used_data":7112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337700,"statfs":{"total":96636764160,"available":96601804800,"internally_reserved":0,"allocated":7282688,"data_stored":8233059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444042,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":6033408,"data_stored":6774883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6553600,"data_stored":8073216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9441280,"data_stored":11612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9961472,"data_stored":12181504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7618560,"data_stored":9445376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8429568,"data_stored":10473472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8728576,"data_stored":10653696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":10989568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4640768,"data_stored":5726208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5517312,"data_stored":6492160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4456448,"data_stored":5419008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6082560,"data_stored":7618560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6766592,"data_stored":7950336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:08.957 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:08.955+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:08.969 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:08.969 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:08.969 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:08.970 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:08.970 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:08.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:09.022 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:09.019+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:09.122 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:09.119+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:09.143 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:09.143 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:09.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:155 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'109, mlcod=24'105)", 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 27.835540482999999, 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30724263200000002, 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:09.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6800000000000002e-07 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306525+0000", 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2228999999999999e-05 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306532+0000", 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7469999999999999e-06 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306539+0000", 2024-04-18T21:52:09.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1250000000000004e-06 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.307270+0000", 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00073067900000000003 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.307309+0000", 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9830000000000003e-05 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613662+0000", 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30635288799999999 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613735+0000", 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2866000000000003e-05 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.148 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:156 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'110, mlcod=24'105)", 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 27.835487709999999, 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30725738200000002, 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306546+0000", 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.16e-07 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306548+0000", 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.672e-06 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306553+0000", 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0730000000000004e-06 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306556+0000", 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.959e-06 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408193+0000", 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101636647 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408269+0000", 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5767999999999997e-05 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613742+0000", 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.205473081 2024-04-18T21:52:09.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613803+0000", 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0965999999999999e-05 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:157 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'111, mlcod=24'105)", 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 27.835472747000001, 2024-04-18T21:52:09.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30730748800000002, 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.72e-07 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306563+0000", 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2730000000000001e-06 2024-04-18T21:52:09.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306572+0000", 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2390000000000008e-06 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306577+0000", 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7709999999999997e-06 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408635+0000", 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102058725 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408654+0000", 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8921000000000001e-05 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613808+0000", 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20515377100000001 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613868+0000", 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9616e-05 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.779555999999999, 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40570043900000002, 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.158 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362478+0000", 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4279999999999999e-06 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362656+0000", 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017710100000000001 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362673+0000", 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7802e-05 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462745+0000", 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100071484 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.767971+0000", 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30522610500000003 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.768177+0000", 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020651900000000001 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.712666764, 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44082012100000001, 2024-04-18T21:52:09.162 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429367+0000", 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5300000000000004e-07 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429643+0000", 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027611500000000001 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429665+0000", 2024-04-18T21:52:09.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1603e-05 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.529793+0000", 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100128809 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868616+0000", 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.33882269300000001 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870186+0000", 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015703480000000001 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.679196893, 2024-04-18T21:52:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40574496100000002, 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0100000000000001e-07 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462933+0000", 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6247000000000002e-05 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462943+0000", 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.944e-06 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462948+0000", 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4979999999999997e-06 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868491+0000", 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40554244699999997 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868581+0000", 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0524000000000003e-05 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.676286488000001, 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71932642099999999, 2024-04-18T21:52:09.170 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9599999999999998e-07 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465774+0000", 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6846e-05 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465781+0000", 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2640000000000002e-06 2024-04-18T21:52:09.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465785+0000", 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7759999999999999e-06 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.185006+0000", 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71922088500000003 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.185073+0000", 2024-04-18T21:52:09.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6853999999999994e-05 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.676217476, 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82169303100000002, 2024-04-18T21:52:09.174 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4000000000000002e-07 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465848+0000", 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1609000000000002e-05 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465853+0000", 2024-04-18T21:52:09.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.366e-06 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465856+0000", 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.151e-06 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286522+0000", 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82066616299999995 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.287509+0000", 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0009863020000000001 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.676120671, 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40453295099999997, 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.178 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465913+0000", 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6199999999999998e-07 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465996+0000", 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3168000000000001e-05 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.466007+0000", 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0662000000000001e-05 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.466014+0000", 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8639999999999998e-06 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870275+0000", 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.404261702 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870445+0000", 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016999300000000001 2024-04-18T21:52:09.181 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.610652571999999, 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.75509254699999995, 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531387+0000", 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2899999999999999e-06 2024-04-18T21:52:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531557+0000", 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016995899999999999 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531588+0000", 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1069999999999999e-05 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531606+0000", 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8479999999999999e-05 2024-04-18T21:52:09.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286385+0000", 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.754778964 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286473+0000", 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7783999999999998e-05 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.575942945, 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858464, 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566092+0000", 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5799999999999999e-06 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566247+0000", 2024-04-18T21:52:09.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155613 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566277+0000", 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0111999999999999e-05 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566315+0000", 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7141999999999998e-05 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.974376+0000", 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.408061795 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.974675+0000", 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029839800000000001 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=7, attrs_read=0))", 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.135266012000001, 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41262958999999999, 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006768+0000", 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4139999999999999e-06 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006798+0000", 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9499e-05 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006823+0000", 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4706999999999999e-05 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006842+0000", 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9507000000000001e-05 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418445+0000", 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41160240300000001 2024-04-18T21:52:09.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419397+0000", 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095206000000000002 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.10s1 41/39 ECSubRead(tid=11, to_read={3:0e754656:::smithi09731216-19:head=0,4096,0}, subchunks={3:0e754656:::smithi09731216-19:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.120442408000001, 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39881851499999998, 2024-04-18T21:52:09.193 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:09.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021616+0000", 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5575000000000002e-05 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021627+0000", 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1099e-05 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.420045+0000", 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39841796699999998 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.420409+0000", 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036387399999999998 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.119791419, 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39759610699999998, 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.197 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6199999999999997e-07 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022253+0000", 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0283e-05 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022265+0000", 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2302e-05 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022271+0000", 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2550000000000003e-06 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419761+0000", 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39748950500000002 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419838+0000", 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7000000000000001e-05 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.119567647, 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39718676600000002, 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022467+0000", 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3030000000000001e-06 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022485+0000", 2024-04-18T21:52:09.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8363999999999999e-05 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022504+0000", 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9283000000000002e-05 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022514+0000", 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8430000000000005e-06 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419513+0000", 2024-04-18T21:52:09.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39699880100000001 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419652+0000", 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013917199999999999 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.119448771, 2024-04-18T21:52:09.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39738200800000001, 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022585+0000", 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7599999999999996e-07 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022601+0000", 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6203000000000001e-05 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022613+0000", 2024-04-18T21:52:09.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1334e-05 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022618+0000", 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4110000000000004e-06 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419896+0000", 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39727772700000002 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419966+0000", 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0556999999999995e-05 2024-04-18T21:52:09.207 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.10s1 41/39 ECSubWriteReply(tid=10, last_complete=41'16, committed=1, applied=1))", 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.119396175, 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39534124999999998, 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:09.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022640+0000", 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0029999999999999e-06 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022652+0000", 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1585e-05 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022656+0000", 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5739999999999997e-06 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.417932+0000", 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39527527800000001 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.417978+0000", 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6350000000000002e-05 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.211 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.10s1 41/39 ECSubWriteReply(tid=10, last_complete=41'16, committed=1, applied=1))", 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.037335877, 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31366916900000003, 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104699+0000", 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.686e-06 2024-04-18T21:52:09.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104713+0000", 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4296e-05 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104742+0000", 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9204999999999999e-05 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104758+0000", 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5254e-05 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418210+0000", 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31345283600000001 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418366+0000", 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155892 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 3.18s6,from:0'0,to:38'13,epoch:41/31,start:3:18000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.035907353000001, 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31261068600000003, 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:09.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106127+0000", 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5000000000000001e-07 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106133+0000", 2024-04-18T21:52:09.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8940000000000004e-06 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106149+0000", 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5534000000000001e-05 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106159+0000", 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.047e-05 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418620+0000", 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31246021499999999 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418658+0000", 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8566000000000003e-05 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418736+0000", 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8157000000000004e-05 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.219 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.020342250000001, 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39824586299999998, 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:09.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121692+0000", 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5499999999999997e-07 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121705+0000", 2024-04-18T21:52:09.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3207000000000001e-05 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121718+0000", 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2913e-05 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121724+0000", 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5030000000000002e-06 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.519849+0000", 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39812503900000001 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.519937+0000", 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7346e-05 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.223 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:09.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:09.225 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:09.225+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:09.326 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:09.323+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:09.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:52:09.396 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:09.397+0000 7f9d75746640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:09.401 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:09.401 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:09.401 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:09.401 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:09.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:09.461 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:09.459+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:09.527 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:09.523+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:09.569 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:09.567+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:09.581 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:09.581 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:09.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:09.628 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:09.629+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:09.695 INFO:tasks.rados.rados.0.smithi097.stdout:146: done (3 left) 2024-04-18T21:52:09.695 INFO:tasks.rados.rados.0.smithi097.stdout:150: done (2 left) 2024-04-18T21:52:09.695 INFO:tasks.rados.rados.0.smithi097.stdout:151: done (1 left) 2024-04-18T21:52:09.695 INFO:tasks.rados.rados.0.smithi097.stdout:153: done (0 left) 2024-04-18T21:52:09.695 INFO:tasks.rados.rados.0.smithi097.stdout:154: snap_create 2024-04-18T21:52:09.719 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 29/26 ECSubWriteReply(tid=2, last_complete=29'3, committed=1, applied=1))", 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.683063011000002, 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20866285400000001, 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034152+0000", 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034153+0000", 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3099999999999999e-07 2024-04-18T21:52:09.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034154+0000", 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.838e-06 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034160+0000", 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3979999999999998e-06 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034179+0000", 2024-04-18T21:52:09.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8751000000000001e-05 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242502+0000", 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.208323065 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242815+0000", 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031357100000000001 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 29/26 ECSubWriteReply(tid=3, last_complete=29'4, committed=1, applied=1))", 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.683028801999999, 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20991810899999999, 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:52:09.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034187+0000", 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5400000000000002e-07 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034189+0000", 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.956e-06 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034196+0000", 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0509999999999999e-06 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.034210+0000", 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4086e-05 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.242874+0000", 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.208664241 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.244105+0000", 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001230521 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:94 3.0s0 3:033c99c3:::smithi09731216-32:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:09.727 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.161365574000001, 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40893576500000001, 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555853+0000", 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1889999999999999e-06 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556617+0000", 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076362200000000004 2024-04-18T21:52:09.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556649+0000", 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2478999999999999e-05 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556665+0000", 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5837000000000001e-05 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556826+0000", 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016137200000000001 2024-04-18T21:52:09.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.557241+0000", 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00041440699999999999 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.559642+0000", 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024011430000000001 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563415+0000", 2024-04-18T21:52:09.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003773233 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964577+0000", 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40116224099999997 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964786+0000", 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000208242 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:95 3.0s0 3:033c99c3:::smithi09731216-32:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.160512477000001, 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858905099999998, 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 95 2024-04-18T21:52:09.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556705+0000", 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6360000000000001e-06 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556715+0000", 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.066e-05 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556734+0000", 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8323999999999999e-05 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556745+0000", 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1240000000000001e-05 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561327+0000", 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0045823330000000001 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561583+0000", 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000255732 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.562607+0000", 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010240990000000001 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563867+0000", 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012605439999999999 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965078+0000", 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40121052699999998 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965292+0000", 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021395600000000001 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:96 3.0s0 3:033c99c3:::smithi09731216-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.160414934999999, 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42323290800000002, 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.738 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 96 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556802+0000", 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7400000000000001e-06 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556808+0000", 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.993e-06 2024-04-18T21:52:09.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556828+0000", 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9364e-05 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.656965+0000", 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10013702300000001 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762039+0000", 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105074903 2024-04-18T21:52:09.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762350+0000", 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031089399999999998 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967780+0000", 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20542997700000001 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.972542+0000", 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004761932 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.979990+0000", 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0074475289999999996 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.980033+0000", 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3553000000000001e-05 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.743 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:222 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.704333345999999, 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30521668800000001, 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.744 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 222 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012885+0000", 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7760000000000002e-06 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012901+0000", 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6563e-05 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012934+0000", 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2719999999999998e-05 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012952+0000", 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7730000000000001e-05 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013179+0000", 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022670500000000001 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:09.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013200+0000", 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1466000000000001e-05 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114024+0000", 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100823616 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114179+0000", 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015544500000000001 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.318099+0000", 2024-04-18T21:52:09.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203919667 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:223 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.704212917, 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40441434900000001, 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.749 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 223 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:09.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013004+0000", 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5400000000000001e-06 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013015+0000", 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0519999999999999e-05 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013077+0000", 2024-04-18T21:52:09.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2848999999999994e-05 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013098+0000", 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1083000000000001e-05 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013359+0000", 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000260408 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013383+0000", 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3867e-05 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114847+0000", 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10146374399999999 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114996+0000", 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149865 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.417417+0000", 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30242047300000002 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:224 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.704082825, 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40564824999999999, 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.754 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 224 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013134+0000", 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.387e-06 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013144+0000", 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0273e-05 2024-04-18T21:52:09.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013166+0000", 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1857000000000001e-05 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013182+0000", 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5645999999999999e-05 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113542+0000", 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10035996599999999 2024-04-18T21:52:09.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113552+0000", 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0492999999999999e-05 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215576+0000", 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102023869 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215730+0000", 2024-04-18T21:52:09.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000153635 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.418781+0000", 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203051124 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:302 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.655564504000001, 2024-04-18T21:52:09.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.406305586, 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 302 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061654+0000", 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061669+0000", 2024-04-18T21:52:09.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4637000000000001e-05 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061699+0000", 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9791e-05 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061714+0000", 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5441999999999999e-05 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061811+0000", 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7153999999999998e-05 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061819+0000", 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9279999999999993e-06 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064181+0000", 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002361447 2024-04-18T21:52:09.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064306+0000", 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012514399999999999 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.065234+0000", 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092848599999999996 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.165102+0000", 2024-04-18T21:52:09.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.099867808000000002 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467877+0000", 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.302775085 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467956+0000", 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9381000000000005e-05 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:303 3.0s0 3:00e42e84:::smithi09731216-41:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.655465114, 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50679023400000001, 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 303 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061751+0000", 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5499999999999998e-07 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061754+0000", 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2339999999999999e-06 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061763+0000", 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3950000000000007e-06 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061769+0000", 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9730000000000002e-06 2024-04-18T21:52:09.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061904+0000", 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013470999999999999 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061911+0000", 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2200000000000003e-06 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163859+0000", 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101947416 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163977+0000", 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117971 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.164692+0000", 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071513499999999999 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267070+0000", 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102378067 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568454+0000", 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.301384333 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568541+0000", 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6224999999999995e-05 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.771 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:304 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.655422642, 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60693648, 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 304 2024-04-18T21:52:09.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3000000000000001e-07 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061797+0000", 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3680000000000002e-06 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061804+0000", 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0809999999999996e-06 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061809+0000", 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5280000000000003e-06 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061950+0000", 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000140529 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061958+0000", 2024-04-18T21:52:09.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5329999999999994e-06 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166016+0000", 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104057925 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166118+0000", 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010206500000000001 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166690+0000", 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057174600000000004 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267073+0000", 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100382727 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668597+0000", 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401524149 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668729+0000", 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132399 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:332 3.0s0 3:00b6e030:::smithi09731216-21:head [copy-from ver 39] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.511263662999999, 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.72786893100000005, 2024-04-18T21:52:09.778 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 332 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205954+0000", 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4370000000000001e-06 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205969+0000", 2024-04-18T21:52:09.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4516e-05 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206001+0000", 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.252e-05 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206022+0000", 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.084e-05 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306220+0000", 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100197961 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306483+0000", 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026249200000000002 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.726594+0000", 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42011143499999998 2024-04-18T21:52:09.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.730261+0000", 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036666060000000002 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933714+0000", 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203453467 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933821+0000", 2024-04-18T21:52:09.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010665700000000001 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:333 3.0s0 3:00b6e030:::smithi09731216-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.51115869, 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52393701999999998, 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:09.784 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 333 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:09.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206058+0000", 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.426e-06 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206065+0000", 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4799999999999998e-06 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206085+0000", 2024-04-18T21:52:09.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0089999999999999e-05 2024-04-18T21:52:09.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206101+0000", 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.643e-05 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306839+0000", 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100738203 2024-04-18T21:52:09.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306987+0000", 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014755599999999999 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729615+0000", 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42262781100000002 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729821+0000", 2024-04-18T21:52:09.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020588799999999999 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729994+0000", 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000173136 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=46, reqid=client.4509.0:360, at_version=47'10, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.8926759200000003, 2024-04-18T21:52:09.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40258300600000002, 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:09.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824542+0000", 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.193e-06 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824777+0000", 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234878 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.794 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824808+0000", 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1633e-05 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924923+0000", 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10011492700000001 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.025179+0000", 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100256004 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.025197+0000", 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8326e-05 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227086+0000", 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201888453 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227122+0000", 2024-04-18T21:52:09.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6591999999999998e-05 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=47, reqid=client.4509.0:365, at_version=47'11, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7922491860000003, 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218968200000002, 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.797 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924967+0000", 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.088e-06 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925088+0000", 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120229 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925104+0000", 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6804e-05 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925115+0000", 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0516999999999999e-05 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.026507+0000", 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101391776 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.026542+0000", 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5126999999999998e-05 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227138+0000", 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20059612399999999 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227156+0000", 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8017e-05 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.801 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=48, reqid=client.4509.0:366, at_version=47'12, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7920815470000004, 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30205703299999997, 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2500000000000001e-07 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925294+0000", 2024-04-18T21:52:09.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016000000000000001 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925308+0000", 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4144999999999999e-05 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925319+0000", 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.008e-05 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.027610+0000", 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102291967 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.027638+0000", 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7453e-05 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227173+0000", 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19953516700000001 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227191+0000", 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7796e-05 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=49, reqid=client.4509.0:367, at_version=47'13, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7918837649999997, 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30187523100000002, 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4700000000000002e-07 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925351+0000", 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8998e-05 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925362+0000", 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1302e-05 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925370+0000", 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6680000000000001e-06 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.028840+0000", 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10347008100000001 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.028867+0000", 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6982999999999999e-05 2024-04-18T21:52:09.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227194+0000", 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19832709500000001 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227207+0000", 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2656999999999999e-05 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.810 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=50, reqid=client.4509.0:361, at_version=47'14, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7918335699999997, 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30184082099999998, 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:09.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6600000000000002e-07 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925457+0000", 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4703000000000006e-05 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925469+0000", 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1932e-05 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925477+0000", 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1529999999999997e-06 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.029353+0000", 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10387573999999999 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.029380+0000", 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7033e-05 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227210+0000", 2024-04-18T21:52:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19783015700000001 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227223+0000", 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2737e-05 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=51, reqid=client.4509.0:362, at_version=47'15, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.942709+0000", 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7745068579999996, 2024-04-18T21:52:09.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28484239, 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942709+0000", 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942709+0000", 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942710+0000", 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4139999999999999e-06 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942750+0000", 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9947999999999998e-05 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942766+0000", 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6575999999999999e-05 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942782+0000", 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5532999999999998e-05 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.226585+0000", 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.283803325 2024-04-18T21:52:09.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.226597+0000", 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1951e-05 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227519+0000", 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092165900000000002 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227551+0000", 2024-04-18T21:52:09.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1983999999999997e-05 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=52, reqid=client.4509.0:363, at_version=47'16, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.943876+0000", 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7733392749999997, 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28622028399999999, 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.820 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943876+0000", 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943876+0000", 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943877+0000", 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3500000000000005e-07 2024-04-18T21:52:09.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943907+0000", 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9966000000000001e-05 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943923+0000", 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6277999999999999e-05 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:09.822 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943985+0000", 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1674000000000006e-05 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227284+0000", 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28329887399999998 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227300+0000", 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6509000000000001e-05 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:09.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.230068+0000", 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002768188 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.230096+0000", 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.796e-05 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:09.824 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:09.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:09.826 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:09.823+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:09.926 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:09.923+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:09.997 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:09.997 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:09.998 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:09.998 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:10.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:10.028 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:10.029+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:10.129 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:10.123+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:10.133 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:10.133 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:10.133 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:10.133 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:10.133 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:10.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:10.230 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:10.227+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:10.312 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:10.312 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:10.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:10.312 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:10.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:83 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 24.260398744, 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28825372900000001, 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 83 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.314 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2e-07 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050828+0000", 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3929999999999998e-06 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050832+0000", 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4909999999999999e-06 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050836+0000", 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5219999999999999e-06 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052455+0000", 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016189710000000001 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052552+0000", 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7213999999999995e-05 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052983+0000", 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043119300000000002 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138342+0000", 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.085359090999999998 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.338942+0000", 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200600308 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339078+0000", 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136026 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:84 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 24.260370086000002, 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59192865299999997, 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 84 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050854+0000", 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1399999999999998e-07 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050855+0000", 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5719999999999999e-06 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050860+0000", 2024-04-18T21:52:10.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050864+0000", 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5750000000000001e-06 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138083+0000", 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087218981000000001 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:10.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138226+0000", 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000143545 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339757+0000", 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20153072 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339856+0000", 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9247999999999996e-05 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642782+0000", 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30292579400000003 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.es0 30/26 ECSubRead(tid=10, to_read={3:77cf8780:::smithi09731216-28:head=0,4096,0}, subchunks={3:77cf8780:::smithi09731216-28:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.971209513000002, 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218228800000002, 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.323 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340093+0000", 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8665000000000007e-05 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440291+0000", 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100198545 2024-04-18T21:52:10.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641845+0000", 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20155352000000001 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642196+0000", 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035155800000000002 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 30/26 ECSubWriteReply(tid=9, last_complete=30'6, committed=1, applied=1))", 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.970527389000001, 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20052165999999999, 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340696+0000", 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340697+0000", 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.004e-06 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340703+0000", 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5260000000000001e-06 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340719+0000", 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.679e-05 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340729+0000", 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9429999999999995e-06 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541183+0000", 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200453357 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.541218+0000", 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5040000000000003e-05 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.870369328999999, 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20088825699999999, 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.330 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440854+0000", 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440856+0000", 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9870000000000002e-06 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440895+0000", 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9325e-05 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440929+0000", 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3293000000000001e-05 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440950+0000", 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0812999999999999e-05 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641679+0000", 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20072974199999999 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641742+0000", 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3096999999999996e-05 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 30/26 ECSubReadReply(tid=10, attrs_read=0))", 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.870011442999999, 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20123138300000001, 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441212+0000", 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441213+0000", 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.057e-06 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.335 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441231+0000", 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7995999999999998e-05 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441251+0000", 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0032999999999999e-05 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.441267+0000", 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5614e-05 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642333+0000", 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201065943 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642443+0000", 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011074 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.cs7 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:52:10.337 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.547668334000001, 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30181939000000002, 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763557+0000", 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.776e-06 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763600+0000", 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3120000000000001e-05 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763635+0000", 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5199e-05 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.863748+0000", 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100112399 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964004+0000", 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10025664500000001 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964030+0000", 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5299999999999998e-05 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065244+0000", 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101214472 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065374+0000", 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000130479 2024-04-18T21:52:10.341 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=19, last_complete=35'11, committed=1, applied=1))", 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 18.033012577000001, 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22495391000000001, 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.342 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278212+0000", 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.004e-06 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278218+0000", 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7490000000000004e-06 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278233+0000", 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5815999999999999e-05 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378308+0000", 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007474600000001 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.502895+0000", 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.124586697 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.503165+0000", 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026989800000000002 2024-04-18T21:52:10.345 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=20, last_complete=35'12, committed=1, applied=1))", 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.932886543999999, 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225695955, 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378338+0000", 2024-04-18T21:52:10.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3499999999999996e-07 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378343+0000", 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0669999999999999e-06 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378356+0000", 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3264e-05 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378367+0000", 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1069e-05 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.603689+0000", 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22532233900000001 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604033+0000", 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034358099999999998 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.349 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=21, last_complete=35'13, committed=1, applied=1))", 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.932846188999999, 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22690856100000001, 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.350 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378378+0000", 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378380+0000", 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7190000000000001e-06 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378389+0000", 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378395+0000", 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5210000000000002e-06 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604185+0000", 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22579012100000001 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.605286+0000", 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011008979999999999 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:230 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.297954839999999, 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.312981385, 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 230 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:10.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013270+0000", 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5300000000000002e-07 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013275+0000", 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-04-18T21:52:10.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013291+0000", 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5466000000000001e-05 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013303+0000", 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2e-05 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013750+0000", 2024-04-18T21:52:10.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044700300000000002 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013767+0000", 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7263999999999999e-05 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225043+0000", 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21127590299999999 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225199+0000", 2024-04-18T21:52:10.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155716 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.326250+0000", 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10105134 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.bs3 36/26 ECSubWrite(tid=36, reqid=client.4509.0:216, at_version=36'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.292864795, 2024-04-18T21:52:10.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20158620099999999, 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018359+0000", 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1299999999999999e-07 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018417+0000", 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7250000000000002e-05 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.018446+0000", 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9567e-05 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118544+0000", 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10009759999999999 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118768+0000", 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022446200000000001 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.118785+0000", 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6286e-05 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.219890+0000", 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10110504400000001 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.219945+0000", 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5278999999999997e-05 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.362 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:315 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 12.218446762999999, 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70415003200000004, 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 315 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092778+0000", 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.725e-06 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.364 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092785+0000", 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7950000000000002e-06 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092802+0000", 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6852999999999999e-05 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092810+0000", 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7950000000000008e-06 2024-04-18T21:52:10.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152681+0000", 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059871316000000001 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152755+0000", 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4179999999999998e-05 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153317+0000", 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00056207900000000005 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253434+0000", 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100116332 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796715+0000", 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54328157899999996 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796927+0000", 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021137799999999999 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:316 3.es0 3:7b3c3586:::smithi09731216-12:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 12.218398019, 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70474535999999999, 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.368 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 316 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092826+0000", 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5999999999999999e-07 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092827+0000", 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714e-06 2024-04-18T21:52:10.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092833+0000", 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0989999999999998e-06 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092836+0000", 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.056e-06 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153954+0000", 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061118211999999998 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154059+0000", 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010548399999999999 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154712+0000", 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065292300000000002 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253436+0000", 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098723985 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797355+0000", 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.543919245 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797571+0000", 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021528199999999999 2024-04-18T21:52:10.373 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:317 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 12.218357616, 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70519442399999999, 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:10.374 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 317 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.375 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000002e-07 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092868+0000", 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1440000000000001e-06 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092873+0000", 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0860000000000001e-06 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192949+0000", 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007585099999999 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252838+0000", 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059889100000000001 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252927+0000", 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8651000000000002e-05 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253260+0000", 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033289600000000001 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.353758+0000", 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100498198 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797947+0000", 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.444189102 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.798060+0000", 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113109 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.379 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=6, reqid=client.4509.0:267, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.605758120000001, 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.305911983, 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:10.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705468+0000", 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4229999999999999e-06 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705683+0000", 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021499900000000001 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705719+0000", 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6758999999999998e-05 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805843+0000", 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10012386299999999 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806004+0000", 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160722 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806040+0000", 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6335999999999999e-05 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011325+0000", 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20528459499999999 2024-04-18T21:52:10.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011377+0000", 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2286e-05 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=8, reqid=client.4509.0:268, at_version=41'15, trim_to=0'0, roll_forward_to=41'12))", 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.505235074, 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20598443299999999, 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.384 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805989+0000", 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5700000000000002e-07 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806154+0000", 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000165575 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806171+0000", 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6946e-05 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806183+0000", 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1394e-05 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.009717+0000", 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203534044 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.009751+0000", 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.455e-05 2024-04-18T21:52:10.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011896+0000", 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002144721 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011973+0000", 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6645999999999998e-05 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=9, reqid=client.4509.0:269, at_version=41'16, trim_to=0'0, roll_forward_to=41'12))", 2024-04-18T21:52:10.388 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.505021688999999, 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.206203308, 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:10.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.37e-07 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806231+0000", 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8955999999999999e-05 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806247+0000", 2024-04-18T21:52:10.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5627000000000001e-05 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.906369+0000", 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100122086 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011040+0000", 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104670897 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011064+0000", 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4219e-05 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.012353+0000", 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012883860000000001 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.012405+0000", 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2500000000000002e-05 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.cs7 51/26 ECSubWrite(tid=66, reqid=osd.5.0:63, at_version=0'0, trim_to=0'0, roll_forward_to=51'31))", 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:07.974844+0000", 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 2.3363792729999999, 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20056485399999999, 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:07.974844+0000", 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:07.974844+0000", 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:07.974845+0000", 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0500000000000002e-07 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:07.974855+0000", 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0019e-05 2024-04-18T21:52:10.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:07.974866+0000", 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1093e-05 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:07.974873+0000", 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3760000000000002e-06 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:07.976654+0000", 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017805500000000001 2024-04-18T21:52:10.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:07.976665+0000", 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1090999999999999e-05 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.175386+0000", 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19872094100000001 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.175409+0000", 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2979000000000001e-05 2024-04-18T21:52:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.19s1 51/31 ECSubWrite(tid=72, reqid=osd.3.0:72, at_version=51'18, trim_to=0'0, roll_forward_to=51'16))", 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:08.067646+0000", 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 2.2435773019999998, 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20140354699999999, 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.067646+0000", 2024-04-18T21:52:10.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.067646+0000", 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.067648+0000", 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1059999999999998e-06 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.067696+0000", 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7707000000000002e-05 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.067726+0000", 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.027e-05 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.067751+0000", 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4346e-05 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.268023+0000", 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20027266499999999 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.268032+0000", 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8400000000000001e-06 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.268979+0000", 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00094734300000000001 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:10.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:08.269050+0000", 2024-04-18T21:52:10.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0270000000000003e-05 2024-04-18T21:52:10.401 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.401 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.401 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.401 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.401 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.401 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:10.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:10.404 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:52:10.406 INFO:tasks.rados.rados.0.smithi097.stdout:154: done (0 left) 2024-04-18T21:52:10.406 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 31 current snap is 11 2024-04-18T21:52:10.406 INFO:tasks.rados.rados.0.smithi097.stdout:155: seq_num 79 ranges {32768=573440,606208=425984,1032192=344064} 2024-04-18T21:52:10.406 INFO:tasks.rados.rados.0.smithi097.stdout:155: writing smithi09731216-31 from 32768 to 606208 tid 1 2024-04-18T21:52:10.406 INFO:tasks.rados.rados.0.smithi097.stdout:155: writing smithi09731216-31 from 606208 to 1032192 tid 2 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:155: writing smithi09731216-31 from 1032192 to 1376256 tid 3 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 16 current snap is 11 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:156: seq_num 80 ranges {0=589824,589824=573440,1163264=196608} 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:156: writing smithi09731216-16 from 0 to 589824 tid 1 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:156: writing smithi09731216-16 from 589824 to 1163264 tid 2 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:156: writing smithi09731216-16 from 1163264 to 1359872 tid 3 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:157: copy_from oid 10 from oid 39 current snap is 11 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:158: read oid 48 snap 7 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:158: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:159: copy_from oid 50 from oid 1 current snap is 11 2024-04-18T21:52:10.407 INFO:tasks.rados.rados.0.smithi097.stdout:160: snap_remove snap 6 2024-04-18T21:52:10.409 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:10.401+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.419 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.420 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.367+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.421 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.422 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.423 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.424 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.426 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.427 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.428 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.371+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.429 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.430 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.431 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.432 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.433 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.434 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.375+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.435 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.436 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.437 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.439 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.379+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:10.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.467+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.467+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.467+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.467+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.467+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.467+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.467+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.467+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.491 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.491 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.491 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.491 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.491 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.491 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.492 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.494 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.495 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.496 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.497 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.498 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.499 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.500 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.479+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.501 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.502 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.503 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.504 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.505 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.506 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.507 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.508 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.483+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.509 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:10.507+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:10.575 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:10.575 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:10.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:402 3.11s0 3:8af06b78:::smithi09731216-10:head [copy-from ver 6 in=55b] snapc c=[c,b,a,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:10.381882+0000", 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.19404517299999999, 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19409000900000001, 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 402 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381882+0000", 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381882+0000", 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381884+0000", 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1490000000000001e-06 2024-04-18T21:52:10.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381893+0000", 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6929999999999996e-06 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381910+0000", 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6977999999999999e-05 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381921+0000", 2024-04-18T21:52:10.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1274000000000001e-05 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.382086+0000", 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016470499999999999 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.382379+0000", 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029282599999999999 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.579 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:403 3.11s0 3:8af06b78:::smithi09731216-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:10.381942+0000", 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.19398496000000001, 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19411626300000001, 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for blocked object", 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 403 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:10.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381942+0000", 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381942+0000", 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381943+0000", 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8100000000000003e-07 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381946+0000", 2024-04-18T21:52:10.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8789999999999999e-06 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381955+0000", 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9879999999999993e-06 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.381960+0000", 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5180000000000004e-06 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:10.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.563516+0000", 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18155606399999999 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:10.563606+0000", 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0055000000000004e-05 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 2 2024-04-18T21:52:10.583 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:10.587 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.583+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.587 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.583+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.587 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.583+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.587 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.583+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:10.637 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:10.635+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:10.651 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:10.651 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:10.669 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:10.670 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:10.686 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:10.687 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.691 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.692 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.693 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.694 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.683+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.695 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.696 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.697 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.699 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.699 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.699 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.699 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.699 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.699 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.699 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.699 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.687+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.703 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:10.703 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:10.711 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.707+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:10.714 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:10.714 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:10.724 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:10.724 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:10.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:10.745 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:10.745 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:10.765 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:10.765 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:10.786 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:10.786 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:10.805 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.806 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.807 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.808 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.809 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.810 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.811 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.812 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.813 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.814 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.815 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.816 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.817 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.787+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.818 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.819 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.820 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.821 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.822 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.823 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.824 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.825 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.826 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.827 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.828 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.829 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.830 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.831 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.832 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.833 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.834 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.835 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.837 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.791+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.838 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.839 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.840 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.841 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.842 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.843 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.844 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:10.795+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:10.845 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:10.845 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:10.845 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:10.846 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:10.846 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:10.845+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:10.863 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:10.864 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:10.870 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:10.870 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:10.884 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:10.888 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:10.888 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:10.947 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:10.945+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:11.017 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:11.017 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:11.031 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:11.076 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:11.075+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:11.150 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:11.147+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:11.155 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:11.155 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:11.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:11.250 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:11.249+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:11.340 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:11.340 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:11.341 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:11.339+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:11.351 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:11.347+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:11.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:11.367 INFO:tasks.rados.rados.0.smithi097.stdout:160: done (5 left) 2024-04-18T21:52:11.367 INFO:tasks.rados.rados.0.smithi097.stdout:161: delete oid 18 current snap is 11 2024-04-18T21:52:11.367 INFO:tasks.rados.rados.0.smithi097.stdout:156: finishing write tid 1 to smithi09731216-16 2024-04-18T21:52:11.367 INFO:tasks.rados.rados.0.smithi097.stdout:156: finishing write tid 2 to smithi09731216-16 2024-04-18T21:52:11.367 INFO:tasks.rados.rados.0.smithi097.stdout:156: finishing write tid 3 to smithi09731216-16 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:159: finishing copy_from racing read to smithi09731216-50 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:156: finishing write tid 4 to smithi09731216-16 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:159: finishing copy_from to smithi09731216-50 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 50 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:157: finishing copy_from racing read to smithi09731216-10 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:156: finishing write tid 5 to smithi09731216-16 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:156: oid 16 updating version 0 to 17 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:156: oid 16 version 17 is already newer than 15 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:156: oid 16 version 17 is already newer than 14 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:156: oid 16 version 17 is already newer than 16 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 16 v 17 (ObjNum 80 snap 11 seq_num 80) dirty exists 2024-04-18T21:52:11.368 INFO:tasks.rados.rados.0.smithi097.stdout:156: left oid 16 (ObjNum 80 snap 11 seq_num 80) 2024-04-18T21:52:11.452 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:11.453+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:11.485 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:11.485 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:11.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:11.543 INFO:tasks.rados.rados.0.smithi097.stdout:157: finishing copy_from to smithi09731216-10 2024-04-18T21:52:11.544 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 10 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-18T21:52:11.553 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:11.551+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:11.622 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:11.622 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:11.636 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:11.654 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:11.651+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:11.679 INFO:tasks.rados.rados.0.smithi097.stdout:156: done (5 left) 2024-04-18T21:52:11.679 INFO:tasks.rados.rados.0.smithi097.stdout:157: done (4 left) 2024-04-18T21:52:11.679 INFO:tasks.rados.rados.0.smithi097.stdout:159: done (3 left) 2024-04-18T21:52:11.679 INFO:tasks.rados.rados.0.smithi097.stdout:161: done (2 left) 2024-04-18T21:52:11.679 INFO:tasks.rados.rados.0.smithi097.stdout:162: snap_remove snap 9 2024-04-18T21:52:11.755 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:11.751+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:11.761 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:11.761 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:11.775 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:11.861 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:11.859+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:11.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:11.941 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:11.956 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:11.957 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:11.957+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:12.012 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:12.007+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:12.013 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:12.011+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:12.013 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:12.011+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:12.058 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:12.057+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:12.118 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:12.118 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:12.132 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:12.177 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:12.175+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:12.214 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:12.211+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:12.214 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:12.211+0000 7efd142f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:12.223 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:12.224 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:12.242 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-04-18T21:52:11.363854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41771552,"num_objects":62,"num_object_clones":11,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":234,"num_read_kb":14151,"num_write":352,"num_write_kb":28434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":536,"ondisk_log_size":536,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":235,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":437788,"kb_used_data":88552,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1132024292,"statfs":{"total":1159641169920,"available":1159192875008,"internally_reserved":0,"allocated":90677248,"data_stored":95881498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":357414205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3612672,"num_objects":-28,"num_object_clones":-25,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":-4,"num_read":4,"num_read_kb":3,"num_write":53,"num_write_kb":3416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009546"},"pg_stats":[{"pgid":"3.19","version":"51'18","reported_seq":153,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.367256+0000","last_change":"2024-04-18T21:52:08.270471+0000","last_active":"2024-04-18T21:52:09.367256+0000","last_peered":"2024-04-18T21:52:09.367256+0000","last_clean":"2024-04-18T21:52:09.367256+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:09.367256+0000","last_undegraded":"2024-04-18T21:52:09.367256+0000","last_fullsized":"2024-04-18T21:52:09.367256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:11.465721+0000","scrub_duration":1415,"objects_trimmed":2,"snaptrim_duration":0.85161646199999996,"stat_sum":{"num_bytes":1376256,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.18","version":"51'20","reported_seq":148,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.467098+0000","last_change":"2024-04-18T21:52:07.964300+0000","last_active":"2024-04-18T21:52:09.467098+0000","last_peered":"2024-04-18T21:52:09.467098+0000","last_clean":"2024-04-18T21:52:09.467098+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:09.467098+0000","last_undegraded":"2024-04-18T21:52:09.467098+0000","last_fullsized":"2024-04-18T21:52:09.467098+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:03.092001+0000","scrub_duration":1983,"objects_trimmed":2,"snaptrim_duration":0.60649774400000001,"stat_sum":{"num_bytes":2523136,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.17","version":"51'8","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369518+0000","last_change":"2024-04-18T21:52:07.711356+0000","last_active":"2024-04-18T21:52:08.369518+0000","last_peered":"2024-04-18T21:52:08.369518+0000","last_clean":"2024-04-18T21:52:08.369518+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:08.369518+0000","last_undegraded":"2024-04-18T21:52:08.369518+0000","last_fullsized":"2024-04-18T21:52:08.369518+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:26.689532+0000","scrub_duration":1063,"objects_trimmed":1,"snaptrim_duration":0.34331814399999999,"stat_sum":{"num_bytes":671744,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.16","version":"32'3","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369067+0000","last_change":"2024-04-18T21:52:07.375837+0000","last_active":"2024-04-18T21:52:08.369067+0000","last_peered":"2024-04-18T21:52:08.369067+0000","last_clean":"2024-04-18T21:52:08.369067+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:08.369067+0000","last_undegraded":"2024-04-18T21:52:08.369067+0000","last_fullsized":"2024-04-18T21:52:08.369067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:22.913391+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.0071241170000000001,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.15","version":"51'9","reported_seq":137,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.568933+0000","last_change":"2024-04-18T21:52:07.967113+0000","last_active":"2024-04-18T21:52:09.568933+0000","last_peered":"2024-04-18T21:52:09.568933+0000","last_clean":"2024-04-18T21:52:09.568933+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:09.568933+0000","last_undegraded":"2024-04-18T21:52:09.568933+0000","last_fullsized":"2024-04-18T21:52:09.568933+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.471986+0000","scrub_duration":811,"objects_trimmed":2,"snaptrim_duration":0.399780054,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.14","version":"51'8","reported_seq":136,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.375667+0000","last_change":"2024-04-18T21:52:07.866739+0000","last_active":"2024-04-18T21:52:10.375667+0000","last_peered":"2024-04-18T21:52:10.375667+0000","last_clean":"2024-04-18T21:52:10.375667+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:10.375667+0000","last_undegraded":"2024-04-18T21:52:10.375667+0000","last_fullsized":"2024-04-18T21:52:10.375667+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.502483+0000","scrub_duration":1515,"objects_trimmed":1,"snaptrim_duration":0.399548444,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.375193+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:10.375193+0000","last_peered":"2024-04-18T21:52:10.375193+0000","last_clean":"2024-04-18T21:52:10.375193+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:10.375193+0000","last_undegraded":"2024-04-18T21:52:10.375193+0000","last_fullsized":"2024-04-18T21:52:10.375193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"32'2","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369232+0000","last_change":"2024-04-18T21:52:07.571117+0000","last_active":"2024-04-18T21:52:08.369232+0000","last_peered":"2024-04-18T21:52:08.369232+0000","last_clean":"2024-04-18T21:52:08.369232+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:08.369232+0000","last_undegraded":"2024-04-18T21:52:08.369232+0000","last_fullsized":"2024-04-18T21:52:08.369232+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.807385+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.101188531,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.470498+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:09.470498+0000","last_peered":"2024-04-18T21:52:09.470498+0000","last_clean":"2024-04-18T21:52:09.470498+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:09.470498+0000","last_undegraded":"2024-04-18T21:52:09.470498+0000","last_fullsized":"2024-04-18T21:52:09.470498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369788+0000","last_change":"2024-04-18T21:52:07.371896+0000","last_active":"2024-04-18T21:52:08.369788+0000","last_peered":"2024-04-18T21:52:08.369788+0000","last_clean":"2024-04-18T21:52:08.369788+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:08.369788+0000","last_undegraded":"2024-04-18T21:52:08.369788+0000","last_fullsized":"2024-04-18T21:52:08.369788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.082419+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.003254673,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.368874+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:09.368874+0000","last_peered":"2024-04-18T21:52:09.368874+0000","last_clean":"2024-04-18T21:52:09.368874+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:09.368874+0000","last_undegraded":"2024-04-18T21:52:09.368874+0000","last_fullsized":"2024-04-18T21:52:09.368874+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":133,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.376242+0000","last_change":"2024-04-18T21:52:07.869046+0000","last_active":"2024-04-18T21:52:10.376242+0000","last_peered":"2024-04-18T21:52:10.376242+0000","last_clean":"2024-04-18T21:52:10.376242+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:10.376242+0000","last_undegraded":"2024-04-18T21:52:10.376242+0000","last_fullsized":"2024-04-18T21:52:10.376242+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.041809+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.40123657299999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1585,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369301+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:08.369301+0000","last_peered":"2024-04-18T21:52:08.369301+0000","last_clean":"2024-04-18T21:52:08.369301+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:08.369301+0000","last_undegraded":"2024-04-18T21:52:08.369301+0000","last_fullsized":"2024-04-18T21:52:08.369301+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"33'7","reported_seq":134,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.569290+0000","last_change":"2024-04-18T21:52:07.968860+0000","last_active":"2024-04-18T21:52:09.569290+0000","last_peered":"2024-04-18T21:52:09.569290+0000","last_clean":"2024-04-18T21:52:09.569290+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:09.569290+0000","last_undegraded":"2024-04-18T21:52:09.569290+0000","last_fullsized":"2024-04-18T21:52:09.569290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:16.707987+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.400066529,"stat_sum":{"num_bytes":835584,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.3","version":"52'74","reported_seq":221,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.371202+0000","last_change":"2024-04-18T21:52:09.105191+0000","last_active":"2024-04-18T21:52:10.371202+0000","last_peered":"2024-04-18T21:52:10.371202+0000","last_clean":"2024-04-18T21:52:10.371202+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:10.371202+0000","last_undegraded":"2024-04-18T21:52:10.371202+0000","last_fullsized":"2024-04-18T21:52:10.371202+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:04.302870+0000","scrub_duration":923,"objects_trimmed":2,"snaptrim_duration":0.50823037299999996,"stat_sum":{"num_bytes":7913472,"num_objects":9,"num_object_clones":2,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.568617+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:09.568617+0000","last_peered":"2024-04-18T21:52:09.568617+0000","last_clean":"2024-04-18T21:52:09.568617+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:09.568617+0000","last_undegraded":"2024-04-18T21:52:09.568617+0000","last_fullsized":"2024-04-18T21:52:09.568617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"51'18","reported_seq":158,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.374179+0000","last_change":"2024-04-18T21:52:07.969018+0000","last_active":"2024-04-18T21:52:10.374179+0000","last_peered":"2024-04-18T21:52:10.374179+0000","last_clean":"2024-04-18T21:52:10.374179+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:10.374179+0000","last_undegraded":"2024-04-18T21:52:10.374179+0000","last_fullsized":"2024-04-18T21:52:10.374179+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.354062+0000","scrub_duration":515,"objects_trimmed":2,"snaptrim_duration":0.60580815700000001,"stat_sum":{"num_bytes":2539520,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.570286+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:09.570286+0000","last_peered":"2024-04-18T21:52:09.570286+0000","last_clean":"2024-04-18T21:52:09.570286+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:09.570286+0000","last_undegraded":"2024-04-18T21:52:09.570286+0000","last_fullsized":"2024-04-18T21:52:09.570286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"51'10","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369001+0000","last_change":"2024-04-18T21:52:07.868892+0000","last_active":"2024-04-18T21:52:08.369001+0000","last_peered":"2024-04-18T21:52:08.369001+0000","last_clean":"2024-04-18T21:52:08.369001+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:08.369001+0000","last_undegraded":"2024-04-18T21:52:08.369001+0000","last_fullsized":"2024-04-18T21:52:08.369001+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.296752+0000","scrub_duration":2059,"objects_trimmed":1,"snaptrim_duration":0.50267342100000001,"stat_sum":{"num_bytes":720896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.361596+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:09.361596+0000","last_peered":"2024-04-18T21:52:09.361596+0000","last_clean":"2024-04-18T21:52:09.361596+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:09.361596+0000","last_undegraded":"2024-04-18T21:52:09.361596+0000","last_fullsized":"2024-04-18T21:52:09.361596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"29'2","reported_seq":135,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.473619+0000","last_change":"2024-04-18T21:52:07.368764+0000","last_active":"2024-04-18T21:52:10.473619+0000","last_peered":"2024-04-18T21:52:10.473619+0000","last_clean":"2024-04-18T21:52:10.473619+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:10.473619+0000","last_undegraded":"2024-04-18T21:52:10.473619+0000","last_fullsized":"2024-04-18T21:52:10.473619+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:31.869645+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.001703574,"stat_sum":{"num_bytes":212992,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.568531+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:09.568531+0000","last_peered":"2024-04-18T21:52:09.568531+0000","last_clean":"2024-04-18T21:52:09.568531+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:09.568531+0000","last_undegraded":"2024-04-18T21:52:09.568531+0000","last_fullsized":"2024-04-18T21:52:09.568531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":251,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.361760+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:09.361760+0000","last_peered":"2024-04-18T21:52:09.361760+0000","last_clean":"2024-04-18T21:52:09.361760+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:09.361760+0000","last_undegraded":"2024-04-18T21:52:09.361760+0000","last_fullsized":"2024-04-18T21:52:09.361760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":126,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.367191+0000","last_change":"2024-04-18T21:52:07.358329+0000","last_active":"2024-04-18T21:52:09.367191+0000","last_peered":"2024-04-18T21:52:09.367191+0000","last_clean":"2024-04-18T21:52:09.367191+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:09.367191+0000","last_undegraded":"2024-04-18T21:52:09.367191+0000","last_fullsized":"2024-04-18T21:52:09.367191+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:13.631171+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.0014568059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.9","version":"29'6","reported_seq":138,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.367477+0000","last_change":"2024-04-18T21:52:08.372007+0000","last_active":"2024-04-18T21:52:09.367477+0000","last_peered":"2024-04-18T21:52:09.367477+0000","last_clean":"2024-04-18T21:52:09.367477+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:09.367477+0000","last_undegraded":"2024-04-18T21:52:09.367477+0000","last_fullsized":"2024-04-18T21:52:09.367477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:17.135455+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.95262314699999995,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.a","version":"51'13","reported_seq":126,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.379716+0000","last_change":"2024-04-18T21:52:07.872122+0000","last_active":"2024-04-18T21:52:10.379716+0000","last_peered":"2024-04-18T21:52:10.379716+0000","last_clean":"2024-04-18T21:52:10.379716+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:10.379716+0000","last_undegraded":"2024-04-18T21:52:10.379716+0000","last_fullsized":"2024-04-18T21:52:10.379716+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:57.927857+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.49812403399999999,"stat_sum":{"num_bytes":1146880,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.b","version":"51'40","reported_seq":151,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.475517+0000","last_change":"2024-04-18T21:52:08.188289+0000","last_active":"2024-04-18T21:52:10.475517+0000","last_peered":"2024-04-18T21:52:10.475517+0000","last_clean":"2024-04-18T21:52:10.475517+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:10.475517+0000","last_undegraded":"2024-04-18T21:52:10.475517+0000","last_fullsized":"2024-04-18T21:52:10.475517+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:45.513752+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.82180306700000005,"stat_sum":{"num_bytes":4833280,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.c","version":"51'37","reported_seq":150,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.569073+0000","last_change":"2024-04-18T21:52:08.595288+0000","last_active":"2024-04-18T21:52:09.569073+0000","last_peered":"2024-04-18T21:52:09.569073+0000","last_clean":"2024-04-18T21:52:09.569073+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:09.569073+0000","last_undegraded":"2024-04-18T21:52:09.569073+0000","last_fullsized":"2024-04-18T21:52:09.569073+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:53.015132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098790132000000003,"stat_sum":{"num_bytes":2605056,"num_objects":5,"num_object_clones":1,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":2008,"num_write":56,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":106,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.462117+0000","last_change":"2024-04-18T21:52:07.462487+0000","last_active":"2024-04-18T21:52:09.462117+0000","last_peered":"2024-04-18T21:52:09.462117+0000","last_clean":"2024-04-18T21:52:09.462117+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:09.462117+0000","last_undegraded":"2024-04-18T21:52:09.462117+0000","last_fullsized":"2024-04-18T21:52:09.462117+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:07.565098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016151869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.e","version":"51'28","reported_seq":143,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.366483+0000","last_change":"2024-04-18T21:52:07.766505+0000","last_active":"2024-04-18T21:52:09.366483+0000","last_peered":"2024-04-18T21:52:09.366483+0000","last_clean":"2024-04-18T21:52:09.366483+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:09.366483+0000","last_undegraded":"2024-04-18T21:52:09.366483+0000","last_fullsized":"2024-04-18T21:52:09.366483+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:02.005586+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.40329519200000002,"stat_sum":{"num_bytes":2785280,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":4039,"num_write":17,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.f","version":"51'13","reported_seq":127,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.362800+0000","last_change":"2024-04-18T21:52:07.813333+0000","last_active":"2024-04-18T21:52:09.362800+0000","last_peered":"2024-04-18T21:52:09.362800+0000","last_clean":"2024-04-18T21:52:09.362800+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:09.362800+0000","last_undegraded":"2024-04-18T21:52:09.362800+0000","last_fullsized":"2024-04-18T21:52:09.362800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:44.167778+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.40760926400000003,"stat_sum":{"num_bytes":1277952,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2885,"num_write":10,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.10","version":"51'20","reported_seq":114,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.373213+0000","last_change":"2024-04-18T21:52:07.969764+0000","last_active":"2024-04-18T21:52:10.373213+0000","last_peered":"2024-04-18T21:52:10.373213+0000","last_clean":"2024-04-18T21:52:10.373213+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:10.373213+0000","last_undegraded":"2024-04-18T21:52:10.373213+0000","last_fullsized":"2024-04-18T21:52:10.373213+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:30.248891+0000","scrub_duration":763,"objects_trimmed":2,"snaptrim_duration":0.59605406900000002,"stat_sum":{"num_bytes":3096576,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.11","version":"51'22","reported_seq":85,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.367096+0000","last_change":"2024-04-18T21:52:08.270803+0000","last_active":"2024-04-18T21:52:09.367096+0000","last_peered":"2024-04-18T21:52:09.367096+0000","last_clean":"2024-04-18T21:52:09.367096+0000","last_became_active":"2024-04-18T21:52:06.594871+0000","last_became_peered":"2024-04-18T21:52:06.594871+0000","last_unstale":"2024-04-18T21:52:09.367096+0000","last_undegraded":"2024-04-18T21:52:09.367096+0000","last_fullsized":"2024-04-18T21:52:09.367096+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:14.360183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.85190311299999999,"stat_sum":{"num_bytes":3235840,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":32,"num_write_kb":3320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,10,1,5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.12","version":"53'7","reported_seq":28,"reported_epoch":53,"state":"active+recovery_wait+undersized+remapped","last_fresh":"2024-04-18T21:52:09.697492+0000","last_change":"2024-04-18T21:52:09.697492+0000","last_active":"2024-04-18T21:52:09.697492+0000","last_peered":"2024-04-18T21:52:09.697492+0000","last_clean":"2024-04-18T21:51:50.031235+0000","last_became_active":"2024-04-18T21:52:08.881396+0000","last_became_peered":"2024-04-18T21:52:08.881396+0000","last_unstale":"2024-04-18T21:52:09.697492+0000","last_undegraded":"2024-04-18T21:52:09.697492+0000","last_fullsized":"2024-04-18T21:52:08.463362+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'5","last_scrub_stamp":"2024-04-18T21:52:01.496294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.496294+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":925696,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,10,0,5,6,1],"acting":[9,7,8,2147483647,0,5,6,1],"avail_no_missing":["9(0)","0(4)","1(6)","1(7)","2(1)","4(3)","5(4)","5(5)","6(2)","6(6)","7(1)","8(2)","8(5)","10(0)","10(3)","11(7)"],"object_location_counts":[{"shards":"0(4),1(7),5(5),6(6),7(1),8(2),9(0)","objects":3}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"51'30","reported_seq":3,"reported_epoch":53,"state":"remapped+peering","last_fresh":"2024-04-18T21:52:07.879241+0000","last_change":"2024-04-18T21:52:09.852883+0000","last_active":"2024-04-18T21:52:09.852883+0000","last_peered":"2024-04-18T21:52:07.879241+0000","last_clean":"2024-04-18T21:52:07.879241+0000","last_became_active":"2024-04-18T21:51:48.486639+0000","last_became_peered":"2024-04-18T21:51:48.486639+0000","last_unstale":"2024-04-18T21:52:07.879241+0000","last_undegraded":"2024-04-18T21:52:07.879241+0000","last_fullsized":"2024-04-18T21:52:07.879241+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.207004411,"stat_sum":{"num_bytes":2220032,"num_objects":7,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":2981,"num_write":15,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4,8,6,11,1,9],"acting":[7,5,4,2147483647,6,2147483647,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":41181184,"num_objects":60,"num_object_clones":11,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":2,"num_read":108,"num_read_kb":14042,"num_write":171,"num_write_kb":25496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":83271680,"data_stored":91668480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378636,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37760,"kb_used_data":6572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334080,"statfs":{"total":96636764160,"available":96598097920,"internally_reserved":0,"allocated":6729728,"data_stored":7333507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378636,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39944,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331896,"statfs":{"total":96636764160,"available":96595861504,"internally_reserved":0,"allocated":8970240,"data_stored":9688456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378637,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39656,"kb_used_data":8468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332184,"statfs":{"total":96636764160,"available":96596156416,"internally_reserved":0,"allocated":8671232,"data_stored":9377411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378636,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40624,"kb_used_data":9440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331216,"statfs":{"total":96636764160,"available":96595165184,"internally_reserved":0,"allocated":9666560,"data_stored":10433928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378636,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38864,"kb_used_data":7684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332976,"statfs":{"total":96636764160,"available":96596967424,"internally_reserved":0,"allocated":7868416,"data_stored":8533406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411339,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":4329472,"data_stored":4203934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411339,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33424,"kb_used_data":6400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338416,"statfs":{"total":96636764160,"available":96602537984,"internally_reserved":0,"allocated":6553600,"data_stored":7169960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411339,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39728,"kb_used_data":8540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332112,"statfs":{"total":96636764160,"available":96596082688,"internally_reserved":0,"allocated":8744960,"data_stored":9151880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36748,"kb_used_data":9728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335092,"statfs":{"total":96636764160,"available":96599134208,"internally_reserved":0,"allocated":9961472,"data_stored":10732958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33936,"kb_used_data":6916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337904,"statfs":{"total":96636764160,"available":96602013696,"internally_reserved":0,"allocated":7081984,"data_stored":7022526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444043,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33516,"kb_used_data":6496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338324,"statfs":{"total":96636764160,"available":96602443776,"internally_reserved":0,"allocated":6651904,"data_stored":6620574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444043,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":5320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":5447680,"data_stored":5612958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6152192,"data_stored":6553600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9416704,"data_stored":10440704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9359360,"data_stored":10383360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7290880,"data_stored":8208384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8069120,"data_stored":9035776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8155136,"data_stored":8818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8380416,"data_stored":9355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5431296,"data_stored":6307840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4870144,"data_stored":5287936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3956736,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6127616,"data_stored":6991872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6062080,"data_stored":6201344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:12.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-18T21:52:12.259 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:12.255+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:12.304 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:12.304 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:12.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:12.360 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:12.359+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:12.407 INFO:tasks.rados.rados.0.smithi097.stdout:162: done (2 left) 2024-04-18T21:52:12.407 INFO:tasks.rados.rados.0.smithi097.stdout:163: snap_create 2024-04-18T21:52:12.407 INFO:tasks.rados.rados.0.smithi097.stdout:155: finishing write tid 1 to smithi09731216-31 2024-04-18T21:52:12.408 INFO:tasks.rados.rados.0.smithi097.stdout:155: finishing write tid 2 to smithi09731216-31 2024-04-18T21:52:12.408 INFO:tasks.rados.rados.0.smithi097.stdout:155: finishing write tid 3 to smithi09731216-31 2024-04-18T21:52:12.408 INFO:tasks.rados.rados.0.smithi097.stdout:155: finishing write tid 4 to smithi09731216-31 2024-04-18T21:52:12.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-18T21:52:12.462 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:12.461+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:12.490 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:12.490 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:12.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:12.562 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:12.559+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:12.611 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:12.611 INFO:teuthology.orchestra.run.smithi097.stdout:{"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":54,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":56,"num_osds":12,"num_up_osds":12,"osd_up_since":1713477089,"num_in_osds":12,"osd_in_since":1713477081,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":33},{"state_name":"active+recovery_wait+undersized+remapped","count":1},{"state_name":"remapped+peering","count":1}],"num_pgs":35,"num_pools":3,"num_objects":62,"data_bytes":41771552,"bytes_used":448294912,"bytes_avail":1159192875008,"bytes_total":1159641169920,"inactive_pgs_ratio":0.02857142873108387,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":204725,"recovering_keys_per_sec":0,"num_objects_recovered":1,"num_bytes_recovered":819200,"num_keys_recovered":0,"read_bytes_sec":767,"write_bytes_sec":874177,"read_op_per_sec":0,"write_op_per_sec":13},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-18T21:51:17.632974+0000","services":{}},"progress_events":{}} 2024-04-18T21:52:12.631 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-04-18T21:52:12.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:12.634 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:12.634 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:12.635 INFO:tasks.rados.rados.0.smithi097.stdout:155: finishing write tid 5 to smithi09731216-31 2024-04-18T21:52:12.635 INFO:tasks.rados.rados.0.smithi097.stdout:155: oid 31 updating version 0 to 34 2024-04-18T21:52:12.635 INFO:tasks.rados.rados.0.smithi097.stdout:155: oid 31 version 34 is already newer than 33 2024-04-18T21:52:12.635 INFO:tasks.rados.rados.0.smithi097.stdout:155: oid 31 version 34 is already newer than 32 2024-04-18T21:52:12.635 INFO:tasks.rados.rados.0.smithi097.stdout:155: oid 31 updating version 34 to 35 2024-04-18T21:52:12.635 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 31 v 35 (ObjNum 79 snap 11 seq_num 79) dirty exists 2024-04-18T21:52:12.635 INFO:tasks.rados.rados.0.smithi097.stdout:155: left oid 31 (ObjNum 79 snap 11 seq_num 79) 2024-04-18T21:52:12.642 INFO:tasks.rados.rados.0.smithi097.stdout:158: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-18T21:52:12.649 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:12.663 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:12.665+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:12.764 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:12.763+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:12.825 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:12.825 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:12.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:12.865 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:12.863+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:12.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:12.926 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:12.927 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:12.945 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-04-18T21:52:12.406867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41771552,"num_objects":62,"num_object_clones":11,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":234,"num_read_kb":14151,"num_write":352,"num_write_kb":28434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":536,"ondisk_log_size":536,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":235,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":437788,"kb_used_data":88552,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1132024292,"statfs":{"total":1159641169920,"available":1159192875008,"internally_reserved":0,"allocated":90677248,"data_stored":95881498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":357414205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3612672,"num_objects":-28,"num_object_clones":-25,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":-4,"num_read":4,"num_read_kb":3,"num_write":53,"num_write_kb":3416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.050338"},"pg_stats":[{"pgid":"3.19","version":"51'18","reported_seq":153,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.367256+0000","last_change":"2024-04-18T21:52:08.270471+0000","last_active":"2024-04-18T21:52:09.367256+0000","last_peered":"2024-04-18T21:52:09.367256+0000","last_clean":"2024-04-18T21:52:09.367256+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:09.367256+0000","last_undegraded":"2024-04-18T21:52:09.367256+0000","last_fullsized":"2024-04-18T21:52:09.367256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:11.465721+0000","scrub_duration":1415,"objects_trimmed":2,"snaptrim_duration":0.85161646199999996,"stat_sum":{"num_bytes":1376256,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.18","version":"51'20","reported_seq":148,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.467098+0000","last_change":"2024-04-18T21:52:07.964300+0000","last_active":"2024-04-18T21:52:09.467098+0000","last_peered":"2024-04-18T21:52:09.467098+0000","last_clean":"2024-04-18T21:52:09.467098+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:09.467098+0000","last_undegraded":"2024-04-18T21:52:09.467098+0000","last_fullsized":"2024-04-18T21:52:09.467098+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:03.092001+0000","scrub_duration":1983,"objects_trimmed":2,"snaptrim_duration":0.60649774400000001,"stat_sum":{"num_bytes":2523136,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.17","version":"51'8","reported_seq":133,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369518+0000","last_change":"2024-04-18T21:52:07.711356+0000","last_active":"2024-04-18T21:52:08.369518+0000","last_peered":"2024-04-18T21:52:08.369518+0000","last_clean":"2024-04-18T21:52:08.369518+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:08.369518+0000","last_undegraded":"2024-04-18T21:52:08.369518+0000","last_fullsized":"2024-04-18T21:52:08.369518+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:26.689532+0000","scrub_duration":1063,"objects_trimmed":1,"snaptrim_duration":0.34331814399999999,"stat_sum":{"num_bytes":671744,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.16","version":"32'3","reported_seq":132,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369067+0000","last_change":"2024-04-18T21:52:07.375837+0000","last_active":"2024-04-18T21:52:08.369067+0000","last_peered":"2024-04-18T21:52:08.369067+0000","last_clean":"2024-04-18T21:52:08.369067+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:08.369067+0000","last_undegraded":"2024-04-18T21:52:08.369067+0000","last_fullsized":"2024-04-18T21:52:08.369067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:22.913391+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.0071241170000000001,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.15","version":"51'9","reported_seq":137,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.568933+0000","last_change":"2024-04-18T21:52:07.967113+0000","last_active":"2024-04-18T21:52:09.568933+0000","last_peered":"2024-04-18T21:52:09.568933+0000","last_clean":"2024-04-18T21:52:09.568933+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:09.568933+0000","last_undegraded":"2024-04-18T21:52:09.568933+0000","last_fullsized":"2024-04-18T21:52:09.568933+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.471986+0000","scrub_duration":811,"objects_trimmed":2,"snaptrim_duration":0.399780054,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.14","version":"51'8","reported_seq":136,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.375667+0000","last_change":"2024-04-18T21:52:07.866739+0000","last_active":"2024-04-18T21:52:10.375667+0000","last_peered":"2024-04-18T21:52:10.375667+0000","last_clean":"2024-04-18T21:52:10.375667+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:10.375667+0000","last_undegraded":"2024-04-18T21:52:10.375667+0000","last_fullsized":"2024-04-18T21:52:10.375667+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.502483+0000","scrub_duration":1515,"objects_trimmed":1,"snaptrim_duration":0.399548444,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":73,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.375193+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:10.375193+0000","last_peered":"2024-04-18T21:52:10.375193+0000","last_clean":"2024-04-18T21:52:10.375193+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:10.375193+0000","last_undegraded":"2024-04-18T21:52:10.375193+0000","last_fullsized":"2024-04-18T21:52:10.375193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"32'2","reported_seq":128,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369232+0000","last_change":"2024-04-18T21:52:07.571117+0000","last_active":"2024-04-18T21:52:08.369232+0000","last_peered":"2024-04-18T21:52:08.369232+0000","last_clean":"2024-04-18T21:52:08.369232+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:08.369232+0000","last_undegraded":"2024-04-18T21:52:08.369232+0000","last_fullsized":"2024-04-18T21:52:08.369232+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.807385+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.101188531,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.470498+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:09.470498+0000","last_peered":"2024-04-18T21:52:09.470498+0000","last_clean":"2024-04-18T21:52:09.470498+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:09.470498+0000","last_undegraded":"2024-04-18T21:52:09.470498+0000","last_fullsized":"2024-04-18T21:52:09.470498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":122,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369788+0000","last_change":"2024-04-18T21:52:07.371896+0000","last_active":"2024-04-18T21:52:08.369788+0000","last_peered":"2024-04-18T21:52:08.369788+0000","last_clean":"2024-04-18T21:52:08.369788+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:08.369788+0000","last_undegraded":"2024-04-18T21:52:08.369788+0000","last_fullsized":"2024-04-18T21:52:08.369788+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.082419+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.003254673,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.368874+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:09.368874+0000","last_peered":"2024-04-18T21:52:09.368874+0000","last_clean":"2024-04-18T21:52:09.368874+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:09.368874+0000","last_undegraded":"2024-04-18T21:52:09.368874+0000","last_fullsized":"2024-04-18T21:52:09.368874+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":133,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.376242+0000","last_change":"2024-04-18T21:52:07.869046+0000","last_active":"2024-04-18T21:52:10.376242+0000","last_peered":"2024-04-18T21:52:10.376242+0000","last_clean":"2024-04-18T21:52:10.376242+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:10.376242+0000","last_undegraded":"2024-04-18T21:52:10.376242+0000","last_fullsized":"2024-04-18T21:52:10.376242+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.041809+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.40123657299999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1585,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369301+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:08.369301+0000","last_peered":"2024-04-18T21:52:08.369301+0000","last_clean":"2024-04-18T21:52:08.369301+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:08.369301+0000","last_undegraded":"2024-04-18T21:52:08.369301+0000","last_fullsized":"2024-04-18T21:52:08.369301+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"33'7","reported_seq":134,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.569290+0000","last_change":"2024-04-18T21:52:07.968860+0000","last_active":"2024-04-18T21:52:09.569290+0000","last_peered":"2024-04-18T21:52:09.569290+0000","last_clean":"2024-04-18T21:52:09.569290+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:09.569290+0000","last_undegraded":"2024-04-18T21:52:09.569290+0000","last_fullsized":"2024-04-18T21:52:09.569290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:16.707987+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.400066529,"stat_sum":{"num_bytes":835584,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.3","version":"52'74","reported_seq":221,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.371202+0000","last_change":"2024-04-18T21:52:09.105191+0000","last_active":"2024-04-18T21:52:10.371202+0000","last_peered":"2024-04-18T21:52:10.371202+0000","last_clean":"2024-04-18T21:52:10.371202+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:10.371202+0000","last_undegraded":"2024-04-18T21:52:10.371202+0000","last_fullsized":"2024-04-18T21:52:10.371202+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:04.302870+0000","scrub_duration":923,"objects_trimmed":2,"snaptrim_duration":0.50823037299999996,"stat_sum":{"num_bytes":7913472,"num_objects":9,"num_object_clones":2,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.568617+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:09.568617+0000","last_peered":"2024-04-18T21:52:09.568617+0000","last_clean":"2024-04-18T21:52:09.568617+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:09.568617+0000","last_undegraded":"2024-04-18T21:52:09.568617+0000","last_fullsized":"2024-04-18T21:52:09.568617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"51'18","reported_seq":158,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.374179+0000","last_change":"2024-04-18T21:52:07.969018+0000","last_active":"2024-04-18T21:52:10.374179+0000","last_peered":"2024-04-18T21:52:10.374179+0000","last_clean":"2024-04-18T21:52:10.374179+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:10.374179+0000","last_undegraded":"2024-04-18T21:52:10.374179+0000","last_fullsized":"2024-04-18T21:52:10.374179+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.354062+0000","scrub_duration":515,"objects_trimmed":2,"snaptrim_duration":0.60580815700000001,"stat_sum":{"num_bytes":2539520,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.570286+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:09.570286+0000","last_peered":"2024-04-18T21:52:09.570286+0000","last_clean":"2024-04-18T21:52:09.570286+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:09.570286+0000","last_undegraded":"2024-04-18T21:52:09.570286+0000","last_fullsized":"2024-04-18T21:52:09.570286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"51'10","reported_seq":140,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-18T21:52:08.369001+0000","last_change":"2024-04-18T21:52:07.868892+0000","last_active":"2024-04-18T21:52:08.369001+0000","last_peered":"2024-04-18T21:52:08.369001+0000","last_clean":"2024-04-18T21:52:08.369001+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:08.369001+0000","last_undegraded":"2024-04-18T21:52:08.369001+0000","last_fullsized":"2024-04-18T21:52:08.369001+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.296752+0000","scrub_duration":2059,"objects_trimmed":1,"snaptrim_duration":0.50267342100000001,"stat_sum":{"num_bytes":720896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.361596+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:09.361596+0000","last_peered":"2024-04-18T21:52:09.361596+0000","last_clean":"2024-04-18T21:52:09.361596+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:09.361596+0000","last_undegraded":"2024-04-18T21:52:09.361596+0000","last_fullsized":"2024-04-18T21:52:09.361596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"29'2","reported_seq":135,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.473619+0000","last_change":"2024-04-18T21:52:07.368764+0000","last_active":"2024-04-18T21:52:10.473619+0000","last_peered":"2024-04-18T21:52:10.473619+0000","last_clean":"2024-04-18T21:52:10.473619+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:10.473619+0000","last_undegraded":"2024-04-18T21:52:10.473619+0000","last_fullsized":"2024-04-18T21:52:10.473619+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:31.869645+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.001703574,"stat_sum":{"num_bytes":212992,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.568531+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:09.568531+0000","last_peered":"2024-04-18T21:52:09.568531+0000","last_clean":"2024-04-18T21:52:09.568531+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:09.568531+0000","last_undegraded":"2024-04-18T21:52:09.568531+0000","last_fullsized":"2024-04-18T21:52:09.568531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":251,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.361760+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:09.361760+0000","last_peered":"2024-04-18T21:52:09.361760+0000","last_clean":"2024-04-18T21:52:09.361760+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:09.361760+0000","last_undegraded":"2024-04-18T21:52:09.361760+0000","last_fullsized":"2024-04-18T21:52:09.361760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":126,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.367191+0000","last_change":"2024-04-18T21:52:07.358329+0000","last_active":"2024-04-18T21:52:09.367191+0000","last_peered":"2024-04-18T21:52:09.367191+0000","last_clean":"2024-04-18T21:52:09.367191+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:09.367191+0000","last_undegraded":"2024-04-18T21:52:09.367191+0000","last_fullsized":"2024-04-18T21:52:09.367191+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:13.631171+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.0014568059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.9","version":"29'6","reported_seq":138,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.367477+0000","last_change":"2024-04-18T21:52:08.372007+0000","last_active":"2024-04-18T21:52:09.367477+0000","last_peered":"2024-04-18T21:52:09.367477+0000","last_clean":"2024-04-18T21:52:09.367477+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:09.367477+0000","last_undegraded":"2024-04-18T21:52:09.367477+0000","last_fullsized":"2024-04-18T21:52:09.367477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:17.135455+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.95262314699999995,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.a","version":"51'13","reported_seq":126,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.379716+0000","last_change":"2024-04-18T21:52:07.872122+0000","last_active":"2024-04-18T21:52:10.379716+0000","last_peered":"2024-04-18T21:52:10.379716+0000","last_clean":"2024-04-18T21:52:10.379716+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:10.379716+0000","last_undegraded":"2024-04-18T21:52:10.379716+0000","last_fullsized":"2024-04-18T21:52:10.379716+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:57.927857+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.49812403399999999,"stat_sum":{"num_bytes":1146880,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.b","version":"51'40","reported_seq":151,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.475517+0000","last_change":"2024-04-18T21:52:08.188289+0000","last_active":"2024-04-18T21:52:10.475517+0000","last_peered":"2024-04-18T21:52:10.475517+0000","last_clean":"2024-04-18T21:52:10.475517+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:10.475517+0000","last_undegraded":"2024-04-18T21:52:10.475517+0000","last_fullsized":"2024-04-18T21:52:10.475517+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:45.513752+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.82180306700000005,"stat_sum":{"num_bytes":4833280,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.c","version":"51'37","reported_seq":150,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.569073+0000","last_change":"2024-04-18T21:52:08.595288+0000","last_active":"2024-04-18T21:52:09.569073+0000","last_peered":"2024-04-18T21:52:09.569073+0000","last_clean":"2024-04-18T21:52:09.569073+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:09.569073+0000","last_undegraded":"2024-04-18T21:52:09.569073+0000","last_fullsized":"2024-04-18T21:52:09.569073+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:53.015132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098790132000000003,"stat_sum":{"num_bytes":2605056,"num_objects":5,"num_object_clones":1,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":2008,"num_write":56,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":106,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.462117+0000","last_change":"2024-04-18T21:52:07.462487+0000","last_active":"2024-04-18T21:52:09.462117+0000","last_peered":"2024-04-18T21:52:09.462117+0000","last_clean":"2024-04-18T21:52:09.462117+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:09.462117+0000","last_undegraded":"2024-04-18T21:52:09.462117+0000","last_fullsized":"2024-04-18T21:52:09.462117+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:07.565098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016151869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.e","version":"51'28","reported_seq":143,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.366483+0000","last_change":"2024-04-18T21:52:07.766505+0000","last_active":"2024-04-18T21:52:09.366483+0000","last_peered":"2024-04-18T21:52:09.366483+0000","last_clean":"2024-04-18T21:52:09.366483+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:09.366483+0000","last_undegraded":"2024-04-18T21:52:09.366483+0000","last_fullsized":"2024-04-18T21:52:09.366483+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:02.005586+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.40329519200000002,"stat_sum":{"num_bytes":2785280,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":4039,"num_write":17,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.f","version":"51'13","reported_seq":127,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.362800+0000","last_change":"2024-04-18T21:52:07.813333+0000","last_active":"2024-04-18T21:52:09.362800+0000","last_peered":"2024-04-18T21:52:09.362800+0000","last_clean":"2024-04-18T21:52:09.362800+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:09.362800+0000","last_undegraded":"2024-04-18T21:52:09.362800+0000","last_fullsized":"2024-04-18T21:52:09.362800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:44.167778+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.40760926400000003,"stat_sum":{"num_bytes":1277952,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2885,"num_write":10,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.10","version":"51'20","reported_seq":114,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-18T21:52:10.373213+0000","last_change":"2024-04-18T21:52:07.969764+0000","last_active":"2024-04-18T21:52:10.373213+0000","last_peered":"2024-04-18T21:52:10.373213+0000","last_clean":"2024-04-18T21:52:10.373213+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:10.373213+0000","last_undegraded":"2024-04-18T21:52:10.373213+0000","last_fullsized":"2024-04-18T21:52:10.373213+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:30.248891+0000","scrub_duration":763,"objects_trimmed":2,"snaptrim_duration":0.59605406900000002,"stat_sum":{"num_bytes":3096576,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.11","version":"51'22","reported_seq":85,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-18T21:52:09.367096+0000","last_change":"2024-04-18T21:52:08.270803+0000","last_active":"2024-04-18T21:52:09.367096+0000","last_peered":"2024-04-18T21:52:09.367096+0000","last_clean":"2024-04-18T21:52:09.367096+0000","last_became_active":"2024-04-18T21:52:06.594871+0000","last_became_peered":"2024-04-18T21:52:06.594871+0000","last_unstale":"2024-04-18T21:52:09.367096+0000","last_undegraded":"2024-04-18T21:52:09.367096+0000","last_fullsized":"2024-04-18T21:52:09.367096+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:14.360183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.85190311299999999,"stat_sum":{"num_bytes":3235840,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":32,"num_write_kb":3320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,10,1,5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.12","version":"53'7","reported_seq":28,"reported_epoch":53,"state":"active+recovery_wait+undersized+remapped","last_fresh":"2024-04-18T21:52:09.697492+0000","last_change":"2024-04-18T21:52:09.697492+0000","last_active":"2024-04-18T21:52:09.697492+0000","last_peered":"2024-04-18T21:52:09.697492+0000","last_clean":"2024-04-18T21:51:50.031235+0000","last_became_active":"2024-04-18T21:52:08.881396+0000","last_became_peered":"2024-04-18T21:52:08.881396+0000","last_unstale":"2024-04-18T21:52:09.697492+0000","last_undegraded":"2024-04-18T21:52:09.697492+0000","last_fullsized":"2024-04-18T21:52:08.463362+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'5","last_scrub_stamp":"2024-04-18T21:52:01.496294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.496294+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":925696,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,10,0,5,6,1],"acting":[9,7,8,2147483647,0,5,6,1],"avail_no_missing":["9(0)","0(4)","1(6)","1(7)","2(1)","4(3)","5(4)","5(5)","6(2)","6(6)","7(1)","8(2)","8(5)","10(0)","10(3)","11(7)"],"object_location_counts":[{"shards":"0(4),1(7),5(5),6(6),7(1),8(2),9(0)","objects":3}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"51'30","reported_seq":3,"reported_epoch":53,"state":"remapped+peering","last_fresh":"2024-04-18T21:52:07.879241+0000","last_change":"2024-04-18T21:52:09.852883+0000","last_active":"2024-04-18T21:52:09.852883+0000","last_peered":"2024-04-18T21:52:07.879241+0000","last_clean":"2024-04-18T21:52:07.879241+0000","last_became_active":"2024-04-18T21:51:48.486639+0000","last_became_peered":"2024-04-18T21:51:48.486639+0000","last_unstale":"2024-04-18T21:52:07.879241+0000","last_undegraded":"2024-04-18T21:52:07.879241+0000","last_fullsized":"2024-04-18T21:52:07.879241+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.207004411,"stat_sum":{"num_bytes":2220032,"num_objects":7,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":2981,"num_write":15,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4,8,6,11,1,9],"acting":[7,5,4,2147483647,6,2147483647,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":41181184,"num_objects":60,"num_object_clones":11,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":2,"num_read":108,"num_read_kb":14042,"num_write":171,"num_write_kb":25496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":83271680,"data_stored":91668480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378636,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37760,"kb_used_data":6572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334080,"statfs":{"total":96636764160,"available":96598097920,"internally_reserved":0,"allocated":6729728,"data_stored":7333507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378636,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39944,"kb_used_data":8760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331896,"statfs":{"total":96636764160,"available":96595861504,"internally_reserved":0,"allocated":8970240,"data_stored":9688456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378637,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39656,"kb_used_data":8468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332184,"statfs":{"total":96636764160,"available":96596156416,"internally_reserved":0,"allocated":8671232,"data_stored":9377411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378636,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40624,"kb_used_data":9440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331216,"statfs":{"total":96636764160,"available":96595165184,"internally_reserved":0,"allocated":9666560,"data_stored":10433928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378636,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38864,"kb_used_data":7684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332976,"statfs":{"total":96636764160,"available":96596967424,"internally_reserved":0,"allocated":7868416,"data_stored":8533406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411339,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":4329472,"data_stored":4203934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411339,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33424,"kb_used_data":6400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338416,"statfs":{"total":96636764160,"available":96602537984,"internally_reserved":0,"allocated":6553600,"data_stored":7169960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411339,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39728,"kb_used_data":8540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332112,"statfs":{"total":96636764160,"available":96596082688,"internally_reserved":0,"allocated":8744960,"data_stored":9151880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36748,"kb_used_data":9728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335092,"statfs":{"total":96636764160,"available":96599134208,"internally_reserved":0,"allocated":9961472,"data_stored":10732958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411338,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33936,"kb_used_data":6916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337904,"statfs":{"total":96636764160,"available":96602013696,"internally_reserved":0,"allocated":7081984,"data_stored":7022526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444043,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33516,"kb_used_data":6496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338324,"statfs":{"total":96636764160,"available":96602443776,"internally_reserved":0,"allocated":6651904,"data_stored":6620574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444043,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":5320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":5447680,"data_stored":5612958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6152192,"data_stored":6553600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9416704,"data_stored":10440704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9359360,"data_stored":10383360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7290880,"data_stored":8208384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8069120,"data_stored":9035776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8155136,"data_stored":8818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8380416,"data_stored":9355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5431296,"data_stored":6307840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4870144,"data_stored":5287936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3956736,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6127616,"data_stored":6991872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6062080,"data_stored":6201344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:12.970 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:12.970 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:12.984 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:12.989 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:12.987+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:13.048 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:13.048 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:13.064 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:13.064 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:13.066 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:13.063+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:13.080 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:13.081 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:13.100 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:13.101 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:13.103 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:13.104 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:13.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:13.121 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:13.141 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:13.141 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:13.161 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:13.161 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:13.169 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:13.169+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:13.182 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:13.183 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:13.203 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:13.204 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:13.225 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:13.225 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:13.244 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:13.245 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:13.249 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:13.249 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:13.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:13.249 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.680552489, 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201960743, 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566449+0000", 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.77e-06 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566495+0000", 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5488999999999998e-05 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566536+0000", 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0756000000000001e-05 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.666638+0000", 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102466 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766859+0000", 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10022078500000001 2024-04-18T21:52:13.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766875+0000", 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5806000000000001e-05 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768312+0000", 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014373389999999999 2024-04-18T21:52:13.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768408+0000", 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6331999999999999e-05 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.4s4 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:52:13.255 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.570025751999999, 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19094267400000001, 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676976+0000", 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9e-06 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677025+0000", 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8730000000000003e-05 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677075+0000", 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0405999999999998e-05 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677101+0000", 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5777000000000001e-05 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.774116+0000", 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.097015016999999995 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.774145+0000", 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9009999999999998e-05 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867877+0000", 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093731420999999995 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867917+0000", 2024-04-18T21:52:13.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0413000000000001e-05 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:118 3.17s0 3:eda737c1:::smithi09731216-40:head [append 0~671744,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.642309260000001, 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1801127490000001, 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.261 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 118 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604693+0000", 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2970000000000002e-06 2024-04-18T21:52:13.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.605995+0000", 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001302276 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606021+0000", 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5320000000000002e-05 2024-04-18T21:52:13.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606036+0000", 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5133000000000001e-05 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606232+0000", 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000196672 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606237+0000", 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7199999999999997e-06 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484281+0000", 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.87804420999999999 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484592+0000", 2024-04-18T21:52:13.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031024299999999998 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.487837+0000", 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003245571 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490891+0000", 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030539719999999998 2024-04-18T21:52:13.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784613+0000", 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.293722237 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784804+0000", 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019009800000000001 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.271 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:119 3.17s0 3:eda737c1:::smithi09731216-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.640927411, 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.179191986, 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:13.272 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 119 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606074+0000", 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.257e-06 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606080+0000", 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0100000000000001e-06 2024-04-18T21:52:13.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606095+0000", 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5413e-05 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606104+0000", 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0899999999999994e-06 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606286+0000", 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000181552 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606287+0000", 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.026e-06 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493720+0000", 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88743291700000004 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493872+0000", 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015190200000000001 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495103+0000", 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00123149 2024-04-18T21:52:13.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.496855+0000", 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017518189999999999 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785096+0000", 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.28824129900000001 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785265+0000", 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016821100000000001 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:120 3.17s0 3:eda737c1:::smithi09731216-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.640858281, 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4107226310000001, 2024-04-18T21:52:13.279 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 120 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606143+0000", 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.16e-07 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606148+0000", 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4569999999999996e-06 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606163+0000", 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503e-05 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606175+0000", 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2036e-05 2024-04-18T21:52:13.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706293+0000", 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100117875 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706297+0000", 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.126e-06 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495621+0000", 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78932349599999996 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495757+0000", 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013642600000000001 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785995+0000", 2024-04-18T21:52:13.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29023745499999998 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.786127+0000", 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013212200000000001 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.016864+0000", 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.23073769199999999 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.285 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=0, last_complete=32'1, committed=1, applied=1))", 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.663182706000001, 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.112466288, 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.286 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583819+0000", 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.604e-06 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583830+0000", 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0883000000000001e-05 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583857+0000", 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7053e-05 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683949+0000", 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100091869 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696264+0000", 2024-04-18T21:52:13.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012315196 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696284+0000", 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9683000000000001e-05 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.7s0,GRANT,e:32,from:3(1),reservation_nonce:3)", 2024-04-18T21:52:13.290 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.597748754000001, 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14739324400000001, 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649253+0000", 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0760000000000001e-06 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649265+0000", 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1523e-05 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649299+0000", 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3662999999999999e-05 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649316+0000", 2024-04-18T21:52:13.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7197000000000002e-05 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796388+0000", 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14707189000000001 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796474+0000", 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6038999999999994e-05 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796645+0000", 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017085599999999999 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.17s0 32/31 ECSubRead(tid=3, to_read={3:eda737c1:::smithi09731216-40:head=0,4096,0}, subchunks={3:eda737c1:::smithi09731216-40:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:13.295 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.459817279999999, 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.210779366, 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:13.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787250+0000", 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7152999999999999e-05 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887353+0000", 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102647 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997704+0000", 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110351217 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997962+0000", 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000258349 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.299 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.17s0 32/31 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.450522276000001, 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200936119, 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796479+0000", 2024-04-18T21:52:13.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8999999999999995e-07 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796491+0000", 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2065000000000001e-05 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796502+0000", 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.116e-05 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796509+0000", 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7170000000000008e-06 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997338+0000", 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20082881799999999 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997414+0000", 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5568999999999994e-05 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.450355556000002, 2024-04-18T21:52:13.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200846881, 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:13.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796645+0000", 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8999999999999998e-07 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796647+0000", 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5239999999999999e-06 2024-04-18T21:52:13.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796655+0000", 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796660+0000", 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4879999999999998e-06 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.307 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997463+0000", 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20080266799999999 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997491+0000", 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8192e-05 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.308 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.449835005000001, 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20052022, 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797166+0000", 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5300000000000004e-07 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797170+0000", 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.233e-06 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797183+0000", 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3454e-05 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797191+0000", 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5769999999999996e-06 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997659+0000", 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20046852900000001 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997685+0000", 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5774000000000001e-05 2024-04-18T21:52:13.312 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.7s0,GRANT,e:32,from:4(6),reservation_nonce:4)", 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.449463191, 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20023016099999999, 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797544+0000", 2024-04-18T21:52:13.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1670000000000002e-06 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797551+0000", 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9959999999999996e-06 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797574+0000", 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.321e-05 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797587+0000", 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.235e-05 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997665+0000", 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200078382 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997695+0000", 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0499999999999999e-05 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997767+0000", 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1556000000000004e-05 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.360008391000001, 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110630505, 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:52:13.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886993+0000", 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.539e-06 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887021+0000", 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7529e-05 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887046+0000", 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4873000000000001e-05 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887060+0000", 2024-04-18T21:52:13.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4324e-05 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997594+0000", 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11053360800000001 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997622+0000", 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8632e-05 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.17s0 32/31 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.887927+0000", 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.359073253999998, 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110115728, 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887927+0000", 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887927+0000", 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887928+0000", 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4399999999999999e-07 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887942+0000", 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4185999999999999e-05 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887957+0000", 2024-04-18T21:52:13.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4951e-05 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887968+0000", 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1735e-05 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.998020+0000", 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110051488 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.998042+0000", 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2524e-05 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.889038+0000", 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.357961718999999, 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.108512738, 2024-04-18T21:52:13.326 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889038+0000", 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.327 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889038+0000", 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889039+0000", 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.63e-07 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889042+0000", 2024-04-18T21:52:13.328 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.241e-06 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889051+0000", 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0389999999999993e-06 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.889059+0000", 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5190000000000003e-06 2024-04-18T21:52:13.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997538+0000", 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10847904 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997551+0000", 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3436e-05 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.330 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/31 ECSubWrite(tid=28, reqid=client.4509.0:304, at_version=41'12, trim_to=0'0, roll_forward_to=32'8))", 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.077632713, 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201742379, 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.331 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.332 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169369+0000", 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6339999999999999e-06 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169413+0000", 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3860999999999999e-05 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169487+0000", 2024-04-18T21:52:13.333 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4578000000000003e-05 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.269583+0000", 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100095613 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369850+0000", 2024-04-18T21:52:13.334 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10026700299999999 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369869+0000", 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9114e-05 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371022+0000", 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001152943 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.335 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371110+0000", 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7633000000000006e-05 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/26 ECSubWrite(tid=56, reqid=osd.10.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'25))", 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:13.336 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.461556689, 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201706881, 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.337 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785445+0000", 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7999999999999999e-06 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.338 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785499+0000", 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3730999999999998e-05 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785543+0000", 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3961000000000001e-05 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785568+0000", 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.561e-05 2024-04-18T21:52:13.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.985970+0000", 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200401511 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.986000+0000", 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0224000000000001e-05 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.340 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987085+0000", 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001085145 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987150+0000", 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4899000000000003e-05 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.341 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.17s0,GRANT,e:42,from:10(3),reservation_nonce:10)", 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 13.966411659, 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200838461, 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:13.342 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280590+0000", 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.587e-06 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280598+0000", 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5450000000000003e-06 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280627+0000", 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8141e-05 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.380804+0000", 2024-04-18T21:52:13.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10017703 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481049+0000", 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10024543 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481088+0000", 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8720999999999997e-05 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.345 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481427+0000", 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000339007 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 42/31 ECSubWrite(tid=30, reqid=client.4509.0:332, at_version=42'14, trim_to=0'0, roll_forward_to=41'12))", 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:59.729136+0000", 2024-04-18T21:52:13.346 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 13.51786355, 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10513689700000001, 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.729136+0000", 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.729136+0000", 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.729138+0000", 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7799999999999999e-06 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.730438+0000", 2024-04-18T21:52:13.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012997390000000001 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.730474+0000", 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5812000000000002e-05 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.830571+0000", 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100097557 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.830747+0000", 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175162 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.830811+0000", 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4648999999999997e-05 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.834211+0000", 2024-04-18T21:52:13.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034001460000000002 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.834273+0000", 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2051999999999995e-05 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 53/51 ECSubWrite(tid=2, reqid=client.4509.0:390, at_version=0'0, trim_to=0'0, roll_forward_to=53'7))", 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:13.351 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 3.5483768270000002, 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201345094, 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698625+0000", 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2869999999999999e-06 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698665+0000", 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9743000000000002e-05 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698719+0000", 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4228999999999999e-05 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.798874+0000", 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100154506 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.799005+0000", 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013080200000000001 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.799030+0000", 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5058e-05 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.899930+0000", 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100900189 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.899968+0000", 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8279999999999999e-05 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.356 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:13.357 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:13.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:13.360 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:13.357+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:13.360 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:13.408 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is unset 2024-04-18T21:52:13.413 INFO:tasks.rados.rados.0.smithi097.stdout:155: done (2 left) 2024-04-18T21:52:13.413 INFO:tasks.rados.rados.0.smithi097.stdout:158: done (1 left) 2024-04-18T21:52:13.413 INFO:tasks.rados.rados.0.smithi097.stdout:163: done (0 left) 2024-04-18T21:52:13.413 INFO:tasks.rados.rados.0.smithi097.stdout:164: read oid 46 snap -1 2024-04-18T21:52:13.414 INFO:tasks.rados.rados.0.smithi097.stdout:164: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-18T21:52:13.414 INFO:tasks.rados.rados.0.smithi097.stdout:165: read oid 7 snap 8 2024-04-18T21:52:13.414 INFO:tasks.rados.rados.0.smithi097.stdout:165: expect (ObjNum 66 snap 4 seq_num 66) 2024-04-18T21:52:13.414 INFO:tasks.rados.rados.0.smithi097.stdout:166: snap_create 2024-04-18T21:52:13.459 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:13.457+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:13.534 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:13.534 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:13.534 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:13.535 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:13.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:13.560 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:13.555+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:13.631 INFO:tasks.rados.rados.0.smithi097.stdout:165: expect (ObjNum 66 snap 4 seq_num 66) 2024-04-18T21:52:13.661 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:13.659+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:13.714 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:13.714 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:13.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:13.762 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:13.759+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:13.836 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:13.831+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=7, reqid=client.4509.0:94, at_version=30'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.188333457999999, 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201169337, 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661077+0000", 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4600000000000002e-06 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661394+0000", 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031720499999999998 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661424+0000", 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9822999999999999e-05 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661441+0000", 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7496e-05 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661661+0000", 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000219602 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661676+0000", 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5333e-05 2024-04-18T21:52:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862187+0000", 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200510299 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862244+0000", 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7119e-05 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.187930151, 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201435581, 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661479+0000", 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0839999999999999e-06 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661509+0000", 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0389999999999999e-05 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661529+0000", 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0219000000000001e-05 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661553+0000", 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3836999999999999e-05 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861851+0000", 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20029786299999999 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861866+0000", 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4769999999999999e-05 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862836+0000", 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097006600000000001 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862913+0000", 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7352000000000001e-05 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.858 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.8s4 30/26 ECSubWrite(tid=12, reqid=client.4509.0:104, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.979849850000001, 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11605781900000001, 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:13.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869560+0000", 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-06 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869593+0000", 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3216000000000002e-05 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869624+0000", 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1164000000000002e-05 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969744+0000", 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100119518 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969958+0000", 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021381800000000001 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969985+0000", 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7376999999999999e-05 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985510+0000", 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015525357 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985616+0000", 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105479 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.17s4 32/31 ECSubWrite(tid=1, reqid=client.4509.0:119, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.354279291000001, 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18978218799999999, 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.863 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495130+0000", 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0860000000000001e-06 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495154+0000", 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4457999999999999e-05 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495174+0000", 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9426999999999999e-05 2024-04-18T21:52:13.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495188+0000", 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4813e-05 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683568+0000", 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.188379871 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683583+0000", 2024-04-18T21:52:13.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5227e-05 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.684845+0000", 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012619560000000001 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.684911+0000", 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5350000000000003e-05 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.14s6 32/31 ECSubWrite(tid=20, reqid=client.4509.0:125, at_version=32'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.252903141000001, 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20154013100000001, 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3799999999999997e-07 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596521+0000", 2024-04-18T21:52:13.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5455999999999999e-05 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596532+0000", 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0881e-05 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696606+0000", 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100073932 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796760+0000", 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10015476600000001 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796771+0000", 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0278999999999999e-05 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797921+0000", 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001150657 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.798045+0000", 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012362199999999999 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.18s4 32/31 ECSubWrite(tid=15, reqid=client.4509.0:137, at_version=32'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.208217+0000", 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.641190675000001, 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101953081, 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.872 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.208217+0000", 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.208217+0000", 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.208218+0000", 2024-04-18T21:52:13.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2e-07 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.208242+0000", 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4759000000000001e-05 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.208261+0000", 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8499999999999999e-05 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.208279+0000", 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8144000000000001e-05 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.309245+0000", 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100966084 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.309263+0000", 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8303000000000001e-05 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310134+0000", 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087051299999999995 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.310170+0000", 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6358e-05 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.876 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 41/31 ECSubWrite(tid=26, reqid=client.4509.0:302, at_version=41'10, trim_to=0'0, roll_forward_to=32'8))", 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.069045+0000", 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.780362846999999, 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106695074, 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069045+0000", 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069045+0000", 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069046+0000", 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0699999999999996e-07 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069106+0000", 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0316000000000003e-05 2024-04-18T21:52:13.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069142+0000", 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5592000000000001e-05 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069190+0000", 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8321e-05 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169610+0000", 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100419604 2024-04-18T21:52:13.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169637+0000", 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7277e-05 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.175645+0000", 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0060080450000000001 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.175740+0000", 2024-04-18T21:52:13.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5211999999999996e-05 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es6 41/26 ECSubWrite(tid=30, reqid=client.4509.0:316, at_version=41'18, trim_to=0'0, roll_forward_to=38'15))", 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.653507286, 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200900518, 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195902+0000", 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3699999999999999e-07 2024-04-18T21:52:13.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195937+0000", 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5682000000000002e-05 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195960+0000", 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2940999999999999e-05 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296055+0000", 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100094687 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296167+0000", 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011265900000000001 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296183+0000", 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5605000000000001e-05 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.396721+0000", 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10053751700000001 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.396801+0000", 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0489999999999997e-05 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.as2 41/26 ECSubWrite(tid=14, reqid=client.4509.0:309, at_version=41'10, trim_to=0'0, roll_forward_to=38'8))", 2024-04-18T21:52:13.885 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.280352+0000", 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.569055467, 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103365524, 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280352+0000", 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280352+0000", 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280354+0000", 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.753e-06 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280592+0000", 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023777800000000001 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280621+0000", 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9166000000000002e-05 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.380731+0000", 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10010949600000001 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.380866+0000", 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013568 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.380892+0000", 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5573999999999999e-05 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.383650+0000", 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027578720000000002 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.383718+0000", 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8205000000000002e-05 2024-04-18T21:52:13.889 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.19s5 41/31 ECSubWrite(tid=38, reqid=client.4509.0:318, at_version=41'9, trim_to=0'0, roll_forward_to=33'7))", 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.474118048999999, 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.199621414, 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375292+0000", 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7069999999999999e-06 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375331+0000", 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9774999999999998e-05 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375361+0000", 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.974e-05 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375379+0000", 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8017e-05 2024-04-18T21:52:13.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375529+0000", 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015042500000000001 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375547+0000", 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7414999999999998e-05 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.574854+0000", 2024-04-18T21:52:13.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19930687499999999 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.574911+0000", 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7460000000000002e-05 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.17s4,RELEASE,e:42,from:8(0),reservation_nonce:0)", 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:13.894 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.062143817000001, 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20038272400000001, 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787265+0000", 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.02e-06 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787274+0000", 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.439e-06 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787295+0000", 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1121e-05 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787312+0000", 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7838000000000001e-05 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987569+0000", 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200256506 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987603+0000", 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4072000000000002e-05 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987647+0000", 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3727999999999999e-05 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.11s3 47/45 ECSubRead(tid=45, to_read={3:8fad7925:::smithi09731216-43:head=0,262144,0}, subchunks={3:8fad7925:::smithi09731216-43:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:13.898 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.451671189000001, 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20153923400000001, 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397739+0000", 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.367e-06 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397761+0000", 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1525000000000001e-05 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397804+0000", 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3488000000000001e-05 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.497972+0000", 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100167611 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.598288+0000", 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100316505 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.599276+0000", 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00098773799999999994 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.901 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.985458+0000", 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.8639502160000001, 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103534346, 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985458+0000", 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985458+0000", 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985459+0000", 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.73e-06 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985910+0000", 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00045051200000000003 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985956+0000", 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6519e-05 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985970+0000", 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3998999999999999e-05 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.088869+0000", 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102898447 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.088992+0000", 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000123139 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.986048+0000", 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.8633595630000004, 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103397726, 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.905 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986048+0000", 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986048+0000", 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986049+0000", 2024-04-18T21:52:13.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9699999999999994e-07 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986623+0000", 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000573541 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986714+0000", 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1388e-05 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986726+0000", 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1609e-05 2024-04-18T21:52:13.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.089309+0000", 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10258281299999999 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.089446+0000", 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013737800000000001 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.908 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.986487+0000", 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.8629211659999996, 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102748308, 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986487+0000", 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986487+0000", 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986488+0000", 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.79e-06 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.987372+0000", 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00088342200000000003 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.987400+0000", 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7827999999999999e-05 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087487+0000", 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100087129 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.089105+0000", 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001617977 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.089235+0000", 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013016199999999999 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.8626908239999995, 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20394542500000001, 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.912 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986718+0000", 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087949+0000", 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10123096600000001 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087978+0000", 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9116999999999999e-05 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188141+0000", 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10016269799999999 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.190504+0000", 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023637250000000001 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.190662+0000", 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000158076 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.188600+0000", 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.6608079440000001, 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103165013, 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188600+0000", 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188600+0000", 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188601+0000", 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.353e-06 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188823+0000", 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00022220700000000001 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188841+0000", 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7308000000000001e-05 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188855+0000", 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4079e-05 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.288469+0000", 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099613660000000007 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291765+0000", 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003296406 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(3.12s0 52/51 [PushOp(3:4c1ede14:::smithi09731216-49:head, version: 33'5, data_included: [0~204800], data_size: 204800, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:4c1ede14:::smithi09731216-49:head@33'5, size: 819200, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 819200, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.5580056759999996, 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10897306900000001, 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291489+0000", 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.721e-05 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.391642+0000", 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100152688 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.397779+0000", 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0061370160000000003 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.400375+0000", 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025961550000000002 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 53/51 ECSubWriteReply(tid=1, last_complete=53'7, committed=1, applied=1))", 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:13.923 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.3311932369999999, 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.180820172, 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:13.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518217+0000", 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9199999999999998e-06 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518227+0000", 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0356e-05 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518255+0000", 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8144e-05 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518274+0000", 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9218000000000001e-05 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.697313+0000", 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17903920600000001 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.699035+0000", 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017213280000000001 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 53/51 ECSubWriteReply(tid=2, last_complete=53'7, committed=1, applied=1))", 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.900106+0000", 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 3.9493022799999999, 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102121302, 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:13.927 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.900106+0000", 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.900106+0000", 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.900106+0000", 2024-04-18T21:52:13.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5300000000000004e-07 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.900111+0000", 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.853e-06 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.900122+0000", 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1122e-05 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.900134+0000", 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2245999999999999e-05 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:10.002134+0000", 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10199918600000001 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:10.002227+0000", 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3342000000000002e-05 2024-04-18T21:52:13.930 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.931 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.931 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.931 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:13.931 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:13.931 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:13.932 INFO:tasks.rados.rados.0.smithi097.stdout:164: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-18T21:52:13.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:13.933 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:13.933+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:13.938 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:13.935+0000 7efd102ea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:14.033 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:14.033+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:14.101 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:14.101 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:14.101 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:14.101 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:14.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:14.134 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:14.131+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:14.235 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:14.235+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:14.283 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:14.283 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:14.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:14.335 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:14.335+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:14.415 INFO:tasks.rados.rados.0.smithi097.stdout:164: done (2 left) 2024-04-18T21:52:14.415 INFO:tasks.rados.rados.0.smithi097.stdout:165: done (1 left) 2024-04-18T21:52:14.415 INFO:tasks.rados.rados.0.smithi097.stdout:166: done (0 left) 2024-04-18T21:52:14.415 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 13 current snap is 13 2024-04-18T21:52:14.415 INFO:tasks.rados.rados.0.smithi097.stdout:167: seq_num 82 ranges {212992=622592,835584=475136,1310720=294912} 2024-04-18T21:52:14.428 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:97 3.4s0 3:209f2930:::smithi09731216-33:head [append 0~540672,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.569059+0000", 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.856890766999999, 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.416082285, 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.429 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.569059+0000", 2024-04-18T21:52:14.430 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.569059+0000", 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.569062+0000", 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2100000000000002e-06 2024-04-18T21:52:14.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.570235+0000", 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011730639999999999 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.570271+0000", 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5876e-05 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.432 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670372+0000", 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10010100199999999 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670552+0000", 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018013600000000001 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670871+0000", 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031866700000000002 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.673680+0000", 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028090709999999998 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.434 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.777705+0000", 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10402547299999999 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985097+0000", 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20739194 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985141+0000", 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3846e-05 2024-04-18T21:52:14.435 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:98 3.4s0 3:209f2930:::smithi09731216-33:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.670413+0000", 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.755536948, 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41727380200000003, 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:14.436 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 98 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670413+0000", 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.437 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670413+0000", 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670414+0000", 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.403e-06 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670425+0000", 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0733e-05 2024-04-18T21:52:14.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670443+0000", 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8427000000000002e-05 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670455+0000", 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1327e-05 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.675388+0000", 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049329830000000002 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.675645+0000", 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000257395 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.440 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676976+0000", 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013308389999999999 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.777778+0000", 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100801826 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087649+0000", 2024-04-18T21:52:14.441 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30987076299999999 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087687+0000", 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8105999999999997e-05 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.442 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:99 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.670540+0000", 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.755409383, 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41733552600000001, 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99 2024-04-18T21:52:14.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670540+0000", 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670540+0000", 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670541+0000", 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6700000000000002e-07 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670548+0000", 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1970000000000001e-06 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670561+0000", 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.381e-05 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.670571+0000", 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3829999999999998e-06 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677632+0000", 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070607589999999998 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677871+0000", 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000239125 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087837+0000", 2024-04-18T21:52:14.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40996610900000002 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087876+0000", 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9175999999999999e-05 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:97 3.4s0 3:209f2930:::smithi09731216-33:head [append 0~540672,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:14.448 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.440094244000001, 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4026191889999999, 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97 2024-04-18T21:52:14.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985858+0000", 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.92e-06 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987540+0000", 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001681463 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987566+0000", 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6043000000000001e-05 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987586+0000", 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9930000000000001e-05 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087950+0000", 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100363824 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087956+0000", 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9200000000000001e-06 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388374+0000", 2024-04-18T21:52:14.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3004186340000001 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388475+0000", 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100455 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:98 3.4s0 3:209f2930:::smithi09731216-33:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.438332842000001, 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.401002842, 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.455 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 98 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:14.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987618+0000", 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5e-06 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987625+0000", 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6769999999999998e-06 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987637+0000", 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1693e-05 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987646+0000", 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3700000000000001e-06 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088140+0000", 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10049390900000001 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088145+0000", 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3600000000000004e-06 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388530+0000", 2024-04-18T21:52:14.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300384269 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388620+0000", 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0063999999999997e-05 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.460 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:99 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.438290102, 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503241456, 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.461 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987661+0000", 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5999999999999991e-07 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987664+0000", 2024-04-18T21:52:14.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6459999999999999e-06 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987675+0000", 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0967e-05 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987684+0000", 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6929999999999996e-06 2024-04-18T21:52:14.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088241+0000", 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100557049 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088244+0000", 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2729999999999998e-06 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388746+0000", 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300502061 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388878+0000", 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131533 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490901+0000", 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102023274 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:117 3.bs0 3:d4cd3926:::smithi09731216-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.238221093, 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.520974363, 2024-04-18T21:52:14.467 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 117 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:14.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187730+0000", 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.127e-06 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187735+0000", 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.536e-06 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187752+0000", 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6186999999999999e-05 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187762+0000", 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0587e-05 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281120+0000", 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093357525999999996 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281296+0000", 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017659300000000001 2024-04-18T21:52:14.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405205+0000", 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.123908465 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405449+0000", 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000244505 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708703+0000", 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30325383700000003 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:126 3.14s0 3:2c76b789:::smithi09731216-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.919692581, 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50118609800000002, 2024-04-18T21:52:14.473 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 126 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506258+0000", 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506267+0000", 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7159999999999992e-06 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506287+0000", 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9542999999999999e-05 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506301+0000", 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4572999999999999e-05 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511483+0000", 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051813989999999997 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511616+0000", 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013376499999999999 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803750+0000", 2024-04-18T21:52:14.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29213323699999999 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803998+0000", 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024861799999999998 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.007443+0000", 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20344494699999999 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.478 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.16s4 32/31 ECSubWrite(tid=12, reqid=client.4509.0:109, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.419507664000001, 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40193662400000002, 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006442+0000", 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006443+0000", 2024-04-18T21:52:14.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999998e-07 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006668+0000", 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000224878 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006685+0000", 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6730000000000001e-05 2024-04-18T21:52:14.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.006703+0000", 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8504000000000001e-05 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206788+0000", 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200085349 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.206807+0000", 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.827e-05 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408321+0000", 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201514675 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408379+0000", 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7414999999999998e-05 2024-04-18T21:52:14.483 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.16s4 32/31 ECSubWrite(tid=15, reqid=client.4509.0:110, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.337793466000001, 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.42111731499999999, 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088157+0000", 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088207+0000", 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0584999999999998e-05 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088241+0000", 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3784e-05 2024-04-18T21:52:14.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088269+0000", 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8314999999999999e-05 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408130+0000", 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31986045299999999 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408140+0000", 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0248000000000001e-05 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509225+0000", 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10108521500000001 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509274+0000", 2024-04-18T21:52:14.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8374999999999997e-05 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:216 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.410037085999999, 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51394664400000001, 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.489 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015916+0000", 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0130000000000002e-06 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015930+0000", 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4705000000000001e-05 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015964+0000", 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4072000000000002e-05 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015996+0000", 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1198999999999997e-05 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016142+0000", 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014630900000000001 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016155+0000", 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2581e-05 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.020038+0000", 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038838789999999998 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.020191+0000", 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000152559 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.021481+0000", 2024-04-18T21:52:14.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012896680000000001 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128155+0000", 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106673879 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.529681+0000", 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40152656199999998 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.529859+0000", 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017821800000000001 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:217 3.bs0 3:d90895f4:::smithi09731216-9:head [rollback 2] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.409900052000001, 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51416292299999999, 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.496 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016051+0000", 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6470000000000001e-06 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016062+0000", 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0451999999999999e-05 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016085+0000", 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2776e-05 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016103+0000", 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8131e-05 2024-04-18T21:52:14.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016278+0000", 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017520200000000001 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016289+0000", 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1048e-05 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.023057+0000", 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067683750000000001 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.023334+0000", 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027651500000000002 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.024976+0000", 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016423379999999999 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128157+0000", 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10318049899999999 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.530073+0000", 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40191629299999998 2024-04-18T21:52:14.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.530213+0000", 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013964700000000001 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:218 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:14.503 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.409816747000001, 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61382267800000001, 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 218 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016134+0000", 2024-04-18T21:52:14.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016141+0000", 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5719999999999999e-06 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016161+0000", 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9814e-05 2024-04-18T21:52:14.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016177+0000", 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6028e-05 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016365+0000", 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018808400000000001 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016384+0000", 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-05 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026326+0000", 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0099418909999999996 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026591+0000", 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026506899999999999 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.027603+0000", 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010121799999999999 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128223+0000", 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100619874 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629817+0000", 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50159395299999998 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629956+0000", 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013901300000000001 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.510 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:298 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~720896] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 16.364479203999998, 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.422019223, 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.511 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 298 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:14.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061474+0000", 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2770000000000001e-06 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063120+0000", 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001645903 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063156+0000", 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.642e-05 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063174+0000", 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7621e-05 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063316+0000", 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000141753 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063330+0000", 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4324999999999999e-05 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.166460+0000", 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103130208 2024-04-18T21:52:14.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.166636+0000", 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175792 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.170069+0000", 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003432778 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279315+0000", 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.109246564 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.483316+0000", 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20400066 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.483490+0000", 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017392199999999999 2024-04-18T21:52:14.517 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:299 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 16.361635022000002, 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.518657543, 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.518 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 299 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064317+0000", 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5900000000000002e-06 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065148+0000", 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000830709 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065184+0000", 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5738999999999998e-05 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065209+0000", 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5627e-05 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065508+0000", 2024-04-18T21:52:14.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029824799999999998 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065539+0000", 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0871999999999999e-05 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.172826+0000", 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.107287676 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.174217+0000", 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001390891 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.177064+0000", 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028466120000000001 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279319+0000", 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102254998 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.582844+0000", 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30352516899999998 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.582972+0000", 2024-04-18T21:52:14.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000128412 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:300 3.bs0 3:d90895f4:::smithi09731216-9:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 16.360690429000002, 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.71857623199999998, 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.525 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 300 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065261+0000", 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9690000000000001e-06 2024-04-18T21:52:14.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065272+0000", 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0545e-05 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065299+0000", 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6927000000000002e-05 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065319+0000", 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0035e-05 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165723+0000", 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100404099 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165747+0000", 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.438e-05 2024-04-18T21:52:14.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279995+0000", 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.114247892 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280276+0000", 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028071299999999998 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281155+0000", 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087942000000000001 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.381976+0000", 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100820639 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783683+0000", 2024-04-18T21:52:14.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40170656199999999 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783836+0000", 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000153051 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:301 3.bs0 3:d90895f4:::smithi09731216-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:14.532 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 16.360558274999999, 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.82307189599999997, 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 301 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065393+0000", 2024-04-18T21:52:14.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.553e-06 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065401+0000", 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9719999999999992e-06 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065425+0000", 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3547e-05 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065447+0000", 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2234000000000001e-05 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165843+0000", 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100395994 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:14.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165860+0000", 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6871000000000002e-05 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281656+0000", 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11579645099999999 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281793+0000", 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013725 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783958+0000", 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50216431100000003 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.784112+0000", 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015411799999999999 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.888463+0000", 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10435159500000001 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:335 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~425984] snapc 6=[6,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:00.073615+0000", 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.352335091, 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41364605900000001, 2024-04-18T21:52:14.539 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 335 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.073615+0000", 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.073615+0000", 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.073618+0000", 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2329999999999998e-06 2024-04-18T21:52:14.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.075142+0000", 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001524091 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.075185+0000", 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3225e-05 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.075215+0000", 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9309999999999999e-05 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.075343+0000", 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000127976 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.075536+0000", 2024-04-18T21:52:14.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019358400000000001 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078315+0000", 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027787720000000001 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.185391+0000", 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.107076481 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.487004+0000", 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30161260000000001 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.487261+0000", 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025678700000000002 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:336 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~573440] snapc 6=[6,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:00.076849+0000", 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.349101031, 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41095928300000001, 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.546 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 336 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.076849+0000", 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.076849+0000", 2024-04-18T21:52:14.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.076852+0000", 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0400000000000001e-06 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.077984+0000", 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011317370000000001 2024-04-18T21:52:14.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078065+0000", 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1619999999999994e-05 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078091+0000", 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6234e-05 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.180314+0000", 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10222298 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.180488+0000", 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017407600000000001 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.182298+0000", 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018095959999999999 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.185393+0000", 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030953059999999999 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.487583+0000", 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30218949299999998 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.487808+0000", 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00022520099999999999 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:337 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~196608] snapc 6=[6,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:00.078161+0000", 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.347788597999999, 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41025571999999999, 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:14.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 337 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078161+0000", 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078161+0000", 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078163+0000", 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2859999999999998e-06 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078532+0000", 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000368754 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078561+0000", 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8507e-05 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.078581+0000", 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0698000000000001e-05 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.183889+0000", 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105307569 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.184007+0000", 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011781399999999999 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.184799+0000", 2024-04-18T21:52:14.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00079178600000000005 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.285553+0000", 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10075487900000001 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.488181+0000", 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20262760399999999 2024-04-18T21:52:14.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:00.488417+0000", 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000235823 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.559 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:14.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:14.563 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:14.561+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:14.563 INFO:tasks.rados.rados.0.smithi097.stdout:167: writing smithi09731216-13 from 212992 to 835584 tid 1 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:167: writing smithi09731216-13 from 835584 to 1310720 tid 2 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:167: writing smithi09731216-13 from 1310720 to 1605632 tid 3 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:168: copy_from oid 15 from oid 5 current snap is 13 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:169: rollback oid 19 current snap is 13 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 19 to 7 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:170: copy_from oid 21 from oid 24 current snap is 13 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:171: copy_from oid 3 from oid 7 current snap is 13 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:172: read oid 33 snap -1 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:172: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 16 current snap is 13 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:173: seq_num 83 ranges {1359872=638976,1998848=622592,2621440=147456} 2024-04-18T21:52:14.564 INFO:tasks.rados.rados.0.smithi097.stdout:173: writing smithi09731216-16 from 1359872 to 1998848 tid 1 2024-04-18T21:52:14.565 INFO:tasks.rados.rados.0.smithi097.stdout:173: writing smithi09731216-16 from 1998848 to 2621440 tid 2 2024-04-18T21:52:14.565 INFO:tasks.rados.rados.0.smithi097.stdout:173: writing smithi09731216-16 from 2621440 to 2768896 tid 3 2024-04-18T21:52:14.565 INFO:tasks.rados.rados.0.smithi097.stdout:174: rmattr oid 34 current snap is 13 2024-04-18T21:52:14.565 INFO:tasks.rados.rados.0.smithi097.stdout:174: done (7 left) 2024-04-18T21:52:14.565 INFO:tasks.rados.rados.0.smithi097.stdout:175: snap_remove snap 11 2024-04-18T21:52:14.662 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:14.659+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.10s3 58/43 ECSubWrite(tid=27, reqid=client.4509.0:427, at_version=58'22, trim_to=0'0, roll_forward_to=51'20))", 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.626793+0000", 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.11202733500000001, 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11206912500000001, 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.737 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.626793+0000", 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.626793+0000", 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.626793+0000", 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7499999999999999e-07 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.626827+0000", 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3454999999999997e-05 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.626842+0000", 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5648999999999999e-05 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.726941+0000", 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.1000988 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727048+0000", 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106727 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727059+0000", 2024-04-18T21:52:14.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0895e-05 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.10s3 58/43 ECSubWrite(tid=28, reqid=client.4509.0:428, at_version=58'23, trim_to=0'0, roll_forward_to=51'20))", 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.726993+0000", 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.01182718, 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011974795, 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.741 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.726993+0000", 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.726993+0000", 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.726994+0000", 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5000000000000001e-07 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727035+0000", 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0357e-05 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727055+0000", 2024-04-18T21:52:14.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0511999999999999e-05 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727068+0000", 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2729e-05 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.10s3 58/43 ECSubWrite(tid=29, reqid=client.4509.0:429, at_version=58'24, trim_to=0'0, roll_forward_to=51'20))", 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.727087+0000", 2024-04-18T21:52:14.744 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.011733380999999999, 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01195747, 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727087+0000", 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727087+0000", 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:14.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727088+0000", 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.44e-07 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727124+0000", 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.693e-05 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:14.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727137+0000", 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2086000000000001e-05 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.727146+0000", 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4499999999999993e-06 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-04-18T21:52:14.747 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:14.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:14.762 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:14.759+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:14.863 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:14.865+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:14.872 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:14.872 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:14.872 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:14.872 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:14.873 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:14.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:14.964 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:14.959+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:15.053 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:15.053 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:15.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:15.053 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:15.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.053 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:108 3.6s0 3:64491ca4:::smithi09731216-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:15.053 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.680309+0000", 2024-04-18T21:52:15.053 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 28.371472567000001, 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29480844099999998, 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 108 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680309+0000", 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680309+0000", 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680311+0000", 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6400000000000001e-06 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680323+0000", 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.146e-05 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680355+0000", 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2149e-05 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.680378+0000", 2024-04-18T21:52:15.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2707000000000001e-05 2024-04-18T21:52:15.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.787116+0000", 2024-04-18T21:52:15.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10673869900000001 2024-04-18T21:52:15.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:15.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.787340+0000", 2024-04-18T21:52:15.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00022391800000000001 2024-04-18T21:52:15.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.975082+0000", 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.187741879 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.975117+0000", 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4989e-05 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.059 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:106 3.6s0 3:64491ca4:::smithi09731216-36:head [append 0~589824,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.963032746, 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.39237912, 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.060 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.061 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 106 2024-04-18T21:52:15.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.061 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:15.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:15.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088750+0000", 2024-04-18T21:52:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3859999999999999e-06 2024-04-18T21:52:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089602+0000", 2024-04-18T21:52:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000852537 2024-04-18T21:52:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089629+0000", 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6066000000000001e-05 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089648+0000", 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9449999999999998e-05 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174464+0000", 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816027000000002 2024-04-18T21:52:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174469+0000", 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5970000000000002e-06 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481023+0000", 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3065548339999999 2024-04-18T21:52:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481128+0000", 2024-04-18T21:52:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010422299999999999 2024-04-18T21:52:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:107 3.6s0 3:64491ca4:::smithi09731216-36:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.962103783, 2024-04-18T21:52:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.301432395, 2024-04-18T21:52:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 107 2024-04-18T21:52:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089679+0000", 2024-04-18T21:52:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.686e-06 2024-04-18T21:52:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089697+0000", 2024-04-18T21:52:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7717e-05 2024-04-18T21:52:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089714+0000", 2024-04-18T21:52:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7433000000000001e-05 2024-04-18T21:52:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089730+0000", 2024-04-18T21:52:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5953000000000001e-05 2024-04-18T21:52:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174547+0000", 2024-04-18T21:52:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816707000000005 2024-04-18T21:52:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174550+0000", 2024-04-18T21:52:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9100000000000001e-06 2024-04-18T21:52:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.983958+0000", 2024-04-18T21:52:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8094078099999999 2024-04-18T21:52:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984087+0000", 2024-04-18T21:52:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012884800000000001 2024-04-18T21:52:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984411+0000", 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032455500000000003 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986825+0000", 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024135609999999998 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391001+0000", 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40417611399999998 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391110+0000", 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109101 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:108 3.6s0 3:64491ca4:::smithi09731216-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.962030137999999, 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4199299170000002, 2024-04-18T21:52:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 108 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089753+0000", 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.46e-06 2024-04-18T21:52:15.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089759+0000", 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5969999999999997e-06 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089782+0000", 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2546999999999999e-05 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089799+0000", 2024-04-18T21:52:15.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6742999999999999e-05 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174653+0000", 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084854180000000001 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.886e-06 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391631+0000", 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2169758150000001 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391728+0000", 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6861000000000007e-05 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509681+0000", 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.117952828 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.085 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:109 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~606208,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.961956584999999, 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4179864210000002, 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089826+0000", 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.147e-06 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090694+0000", 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00086783200000000002 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090726+0000", 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2379999999999998e-05 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090746+0000", 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9466999999999998e-05 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174599+0000", 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083853091000000005 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174600+0000", 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.097e-06 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985378+0000", 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.810778236 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985469+0000", 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0805999999999998e-05 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986312+0000", 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084315300000000002 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089266+0000", 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10295370099999999 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507611+0000", 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41834516199999999 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507811+0000", 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000200349 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:110 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.961013181999999, 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.429779607, 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 110 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090769+0000", 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-04-18T21:52:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090782+0000", 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2547e-05 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090797+0000", 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5109000000000001e-05 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090807+0000", 2024-04-18T21:52:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8670000000000006e-06 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083849218000000003 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174657+0000", 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8700000000000004e-07 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087432+0000", 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9127752569999998 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087519+0000", 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6668000000000004e-05 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088034+0000", 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00051538599999999995 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089552+0000", 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001518295 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520465+0000", 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.43091233000000001 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520548+0000", 2024-04-18T21:52:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2947000000000004e-05 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:111 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.960963443000001, 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5330603059999999, 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 111 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090819+0000", 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6000000000000003e-07 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090822+0000", 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.066e-06 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090834+0000", 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2486999999999999e-05 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090844+0000", 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9899999999999992e-06 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275049+0000", 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.184205065 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275055+0000", 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4389999999999999e-06 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088362+0000", 2024-04-18T21:52:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8133068830000001 2024-04-18T21:52:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088621+0000", 2024-04-18T21:52:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025974499999999997 2024-04-18T21:52:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520968+0000", 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.432346972 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521091+0000", 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122263 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.623878+0000", 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102787636 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:127 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~720896,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.237488458000001, 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50504046999999996, 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.107 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 127 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:15.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814297+0000", 2024-04-18T21:52:15.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3120000000000004e-06 2024-04-18T21:52:15.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.110 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815282+0000", 2024-04-18T21:52:15.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0009845469999999999 2024-04-18T21:52:15.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815306+0000", 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4488999999999999e-05 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815321+0000", 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4365999999999999e-05 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901001+0000", 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085680191000000003 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901126+0000", 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125506 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902346+0000", 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012201989999999999 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904676+0000", 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023299459999999998 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319226+0000", 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.414549324 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319333+0000", 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010759 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.113 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:128 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.236433701999999, 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70473296200000002, 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.114 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 128 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815349+0000", 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.003e-06 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815353+0000", 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5700000000000003e-06 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815362+0000", 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4540000000000007e-06 2024-04-18T21:52:15.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815369+0000", 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4059999999999999e-06 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903371+0000", 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088001653999999999 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903511+0000", 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000139889 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904181+0000", 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067014400000000001 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.905274+0000", 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010929430000000001 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519876+0000", 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61460184500000004 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520081+0000", 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020505400000000001 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.119 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.1s0,GRANT,e:32,from:5(4),reservation_nonce:5)", 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.150247451999999, 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61973115499999998, 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:15.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:15.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901535+0000", 2024-04-18T21:52:15.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.206e-06 2024-04-18T21:52:15.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901541+0000", 2024-04-18T21:52:15.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.331e-06 2024-04-18T21:52:15.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901559+0000", 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7977000000000001e-05 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.001690+0000", 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100130964 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521089+0000", 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51939875099999999 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521142+0000", 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3324e-05 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521265+0000", 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122602 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.125 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:130 3.1s0 3:87c089d9:::smithi09731216-44:head [append 0~737280,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.140775182999999, 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61075046099999997, 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.126 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 130 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911008+0000", 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.356e-06 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911859+0000", 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085008399999999997 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911877+0000", 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8147000000000001e-05 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911886+0000", 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012129+0000", 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100243058 2024-04-18T21:52:15.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012391+0000", 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026173100000000002 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.014631+0000", 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002240344 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118257+0000", 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103625859 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521564+0000", 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40330724200000001 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521757+0000", 2024-04-18T21:52:15.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019236000000000001 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:131 3.1s0 3:87c089d9:::smithi09731216-44:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.139876873999999, 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70851564899999997, 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:15.132 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 131 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:15.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911905+0000", 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9800000000000002e-07 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911910+0000", 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4499999999999997e-06 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911919+0000", 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911927+0000", 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5700000000000004e-06 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017049+0000", 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105122775 2024-04-18T21:52:15.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017158+0000", 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010845600000000001 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017640+0000", 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00048220899999999999 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118261+0000", 2024-04-18T21:52:15.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100621063 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620272+0000", 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50201088400000005 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620420+0000", 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000148064 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 32/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.032012218999999, 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29966542299999999, 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019772+0000", 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2869999999999999e-06 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019780+0000", 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8990000000000001e-06 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019783+0000", 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8460000000000001e-06 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.140 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319409+0000", 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29962544499999999 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319435+0000", 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5605999999999998e-05 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:15.141 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.029753426999999, 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.297022972, 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022030+0000", 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7940000000000001e-06 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022039+0000", 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6339999999999996e-06 2024-04-18T21:52:15.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022063+0000", 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4026e-05 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022074+0000", 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0705e-05 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319012+0000", 2024-04-18T21:52:15.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29693840300000002 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319051+0000", 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.841e-05 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 33/31 ECSubWriteReply(tid=15, last_complete=32'3, committed=1, applied=1))", 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.962324810999998, 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30109323399999999, 2024-04-18T21:52:15.145 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089458+0000", 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5859999999999999e-06 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089467+0000", 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0420000000000008e-06 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089492+0000", 2024-04-18T21:52:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5211999999999998e-05 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.189619+0000", 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100126885 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.390510+0000", 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20089099299999999 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.390550+0000", 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9515999999999999e-05 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.944892432, 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31260053300000001, 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106890+0000", 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8599999999999996e-07 2024-04-18T21:52:15.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106896+0000", 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9139999999999996e-06 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106911+0000", 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4800999999999999e-05 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106920+0000", 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9379999999999998e-06 2024-04-18T21:52:15.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419470+0000", 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.312550204 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419489+0000", 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.969e-05 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:15.152 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.846967659000001, 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.316195273, 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:15.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1699999999999998e-07 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204818+0000", 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5659999999999998e-06 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204828+0000", 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0478e-05 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204836+0000", 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.413e-06 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519654+0000", 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.314818722 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521009+0000", 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001354577 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=13, last_complete=32'5, committed=1, applied=1))", 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.846933973999999, 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31487986099999998, 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.156 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204848+0000", 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8299999999999998e-07 2024-04-18T21:52:15.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204850+0000", 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6429999999999999e-06 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204859+0000", 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0550000000000005e-06 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204866+0000", 2024-04-18T21:52:15.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5320000000000003e-06 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519669+0000", 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31480259199999999 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519727+0000", 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8656000000000002e-05 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.846901117000002, 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31508690700000003, 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:15.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204881+0000", 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3000000000000002e-07 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204884+0000", 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.976e-06 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204900+0000", 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6418000000000001e-05 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204908+0000", 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3080000000000008e-06 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519923+0000", 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31501511199999999 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519967+0000", 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3763000000000001e-05 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:15.163 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.84415194, 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31221361399999997, 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:15.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:15.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:15.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:15.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207631+0000", 2024-04-18T21:52:15.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.42e-06 2024-04-18T21:52:15.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207640+0000", 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7350000000000002e-06 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207668+0000", 2024-04-18T21:52:15.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8881999999999999e-05 2024-04-18T21:52:15.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207686+0000", 2024-04-18T21:52:15.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7136999999999999e-05 2024-04-18T21:52:15.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519798+0000", 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31211196099999999 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519843+0000", 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5479000000000003e-05 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:15.168 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:15.170 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:15.169+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:15.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:15.271 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:15.269+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:15.302 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:15.302 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:15.302 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:15.302 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:15.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:15.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:15.372 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:15.367+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:15.432 INFO:tasks.rados.rados.0.smithi097.stdout:175: done (7 left) 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:167: finishing write tid 1 to smithi09731216-13 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:167: finishing write tid 2 to smithi09731216-13 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:169: finishing rollback tid 0 to smithi09731216-19 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:169: finishing rollback tid 1 to smithi09731216-19 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:168: finishing copy_from racing read to smithi09731216-15 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:173: finishing write tid 1 to smithi09731216-16 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:169: finishing rollback tid 2 to smithi09731216-19 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 19 v 24 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:171: finishing copy_from racing read to smithi09731216-3 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:170: finishing copy_from racing read to smithi09731216-21 2024-04-18T21:52:15.433 INFO:tasks.rados.rados.0.smithi097.stdout:172: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-18T21:52:15.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-18T21:52:15.438 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:15.438 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:15.438 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:15.438 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:15.438 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:15.448 INFO:tasks.rados.rados.0.smithi097.stdout:167: finishing write tid 3 to smithi09731216-13 2024-04-18T21:52:15.448 INFO:tasks.rados.rados.0.smithi097.stdout:176: read oid 39 snap -1 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:176: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:169: done (7 left) 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:172: done (6 left) 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:177: rmattr oid 12 current snap is 13 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:177: done (6 left) 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:178: snap_create 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:167: finishing write tid 4 to smithi09731216-13 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:173: finishing write tid 2 to smithi09731216-16 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:170: finishing copy_from to smithi09731216-21 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 21 v 39 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:168: finishing copy_from to smithi09731216-15 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 15 v 14 (ObjNum 77 snap 10 seq_num 77) dirty exists 2024-04-18T21:52:15.449 INFO:tasks.rados.rados.0.smithi097.stdout:171: finishing copy_from to smithi09731216-3 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 3 v 11 (ObjNum 66 snap 4 seq_num 66) dirty exists 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:173: finishing write tid 3 to smithi09731216-16 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:167: finishing write tid 5 to smithi09731216-13 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:167: oid 13 updating version 0 to 6 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:167: oid 13 version 6 is already newer than 5 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:167: oid 13 updating version 6 to 7 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:167: oid 13 version 7 is already newer than 4 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 13 v 7 (ObjNum 82 snap 13 seq_num 82) dirty exists 2024-04-18T21:52:15.450 INFO:tasks.rados.rados.0.smithi097.stdout:167: left oid 13 (ObjNum 82 snap 13 seq_num 82) 2024-04-18T21:52:15.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:15.501 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:15.499+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:15.573 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:15.571+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:15.598 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:15.598 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:15.644 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:15.644 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:15.652 INFO:tasks.rados.rados.0.smithi097.stdout:173: finishing write tid 4 to smithi09731216-16 2024-04-18T21:52:15.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:15.674 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:15.673+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:15.775 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:15.771+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:15.779 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:15.779 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:15.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 59/31 ECSubWriteReply(tid=37, last_complete=59'22, committed=1, applied=1))", 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:15.757182+0000", 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.022886370999999999, 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.022922207, 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.757182+0000", 2024-04-18T21:52:15.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.757182+0000", 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.757183+0000", 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9999999999999996e-07 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.757189+0000", 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4029999999999998e-06 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.757205+0000", 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6646999999999998e-05 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.757216+0000", 2024-04-18T21:52:15.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1032999999999999e-05 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 59/31 ECSubWriteReply(tid=37, last_complete=59'22, committed=1, applied=1))", 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:15.758414+0000", 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.021654820000000002, 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.021772218999999999, 2024-04-18T21:52:15.783 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.758414+0000", 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.758414+0000", 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:15.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.758415+0000", 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1699999999999997e-07 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.758419+0000", 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4859999999999999e-06 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.758433+0000", 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3601000000000001e-05 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.758441+0000", 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1880000000000002e-06 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 59/31 ECSubWriteReply(tid=37, last_complete=59'22, committed=1, applied=1))", 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:15.663855+0000", 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.116213861, 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116404656, 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.663855+0000", 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:15.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.663855+0000", 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.663855+0000", 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6000000000000004e-07 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:15.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.663859+0000", 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9920000000000001e-06 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.663869+0000", 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1239999999999994e-06 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.663878+0000", 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0270000000000001e-06 2024-04-18T21:52:15.789 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:15.790 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:15.790 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:15.790 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:15.790 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:15.790 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 3 2024-04-18T21:52:15.790 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:15.790 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:15.791 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:15.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:15.807 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:15.808 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:15.825 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:15.845 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:15.867 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:15.867 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:15.875 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:15.871+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:15.896 INFO:tasks.rados.rados.0.smithi097.stdout:173: finishing write tid 5 to smithi09731216-16 2024-04-18T21:52:15.896 INFO:tasks.rados.rados.0.smithi097.stdout:173: oid 16 updating version 0 to 21 2024-04-18T21:52:15.896 INFO:tasks.rados.rados.0.smithi097.stdout:173: oid 16 updating version 21 to 22 2024-04-18T21:52:15.896 INFO:tasks.rados.rados.0.smithi097.stdout:173: oid 16 version 22 is already newer than 19 2024-04-18T21:52:15.896 INFO:tasks.rados.rados.0.smithi097.stdout:173: oid 16 version 22 is already newer than 20 2024-04-18T21:52:15.896 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 16 v 22 (ObjNum 83 snap 13 seq_num 83) dirty exists 2024-04-18T21:52:15.897 INFO:tasks.rados.rados.0.smithi097.stdout:173: left oid 16 (ObjNum 83 snap 13 seq_num 83) 2024-04-18T21:52:15.897 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:15.891+0000 7efd0e2e6640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap d after trimming was started 2024-04-18T21:52:15.922 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:15.922 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:15.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:15.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:15.972 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:15.972 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:16.000 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:16.000 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:15.995+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:16.026 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:16.026 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:16.057 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:16.058 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:16.077 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:16.075+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:16.083 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:16.083 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:16.093 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:16.093 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:16.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:16.109 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:16.109 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:16.154 INFO:tasks.rados.rados.0.smithi097.stdout:176: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-18T21:52:16.197 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:16.195+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:16.269 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:16.270 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:16.279 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:16.275+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:16.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:16.286 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:16.287 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:16.305 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-04-18T21:52:16.258328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51790368,"num_objects":73,"num_object_clones":21,"num_object_copies":572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":3,"num_read":253,"num_read_kb":21452,"num_write":342,"num_write_kb":34250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2539520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":239,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":451920,"kb_used_data":102664,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1132010160,"statfs":{"total":1159641169920,"available":1159178403840,"internally_reserved":0,"allocated":105127936,"data_stored":114421825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":357414205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,1,1,2,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10018816,"num_objects":11,"num_object_clones":10,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":-6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":19,"num_read_kb":7301,"num_write":-10,"num_write_kb":5816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001119"},"pg_stats":[{"pgid":"3.19","version":"56'22","reported_seq":181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.627044+0000","last_change":"2024-04-18T21:52:12.990105+0000","last_active":"2024-04-18T21:52:14.627044+0000","last_peered":"2024-04-18T21:52:14.627044+0000","last_clean":"2024-04-18T21:52:14.627044+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:14.627044+0000","last_undegraded":"2024-04-18T21:52:14.627044+0000","last_fullsized":"2024-04-18T21:52:14.627044+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.552392+0000","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0.43035931799999999,"stat_sum":{"num_bytes":1376256,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.18","version":"51'20","reported_seq":174,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.526784+0000","last_change":"2024-04-18T21:52:12.561250+0000","last_active":"2024-04-18T21:52:14.526784+0000","last_peered":"2024-04-18T21:52:14.526784+0000","last_clean":"2024-04-18T21:52:14.526784+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:14.526784+0000","last_undegraded":"2024-04-18T21:52:14.526784+0000","last_fullsized":"2024-04-18T21:52:14.526784+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:07.165130+0000","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0.10117693699999999,"stat_sum":{"num_bytes":2523136,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2258,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.17","version":"51'8","reported_seq":155,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.623152+0000","last_change":"2024-04-18T21:52:12.486846+0000","last_active":"2024-04-18T21:52:13.623152+0000","last_peered":"2024-04-18T21:52:13.623152+0000","last_clean":"2024-04-18T21:52:13.623152+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:13.623152+0000","last_undegraded":"2024-04-18T21:52:13.623152+0000","last_fullsized":"2024-04-18T21:52:13.623152+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:19.410147+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.0026122879999999999,"stat_sum":{"num_bytes":671744,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.16","version":"32'3","reported_seq":154,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.521746+0000","last_change":"2024-04-18T21:52:12.428844+0000","last_active":"2024-04-18T21:52:13.521746+0000","last_peered":"2024-04-18T21:52:13.521746+0000","last_clean":"2024-04-18T21:52:13.521746+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:13.521746+0000","last_undegraded":"2024-04-18T21:52:13.521746+0000","last_fullsized":"2024-04-18T21:52:13.521746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.631013+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.00193593,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.15","version":"51'9","reported_seq":160,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.417338+0000","last_change":"2024-04-18T21:52:12.822985+0000","last_active":"2024-04-18T21:52:14.417338+0000","last_peered":"2024-04-18T21:52:14.417338+0000","last_clean":"2024-04-18T21:52:14.417338+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:14.417338+0000","last_undegraded":"2024-04-18T21:52:14.417338+0000","last_fullsized":"2024-04-18T21:52:14.417338+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:56.919539+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.30495551599999998,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.14","version":"51'8","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.455291+0000","last_change":"2024-04-18T21:52:15.456411+0000","last_active":"2024-04-18T21:52:15.455291+0000","last_peered":"2024-04-18T21:52:15.455291+0000","last_clean":"2024-04-18T21:52:15.455291+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:15.455291+0000","last_undegraded":"2024-04-18T21:52:15.455291+0000","last_fullsized":"2024-04-18T21:52:15.455291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.605862+0000","scrub_duration":1515,"objects_trimmed":0,"snaptrim_duration":0.0017272139999999999,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.459239+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:15.459239+0000","last_peered":"2024-04-18T21:52:15.459239+0000","last_clean":"2024-04-18T21:52:15.459239+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:15.459239+0000","last_undegraded":"2024-04-18T21:52:15.459239+0000","last_fullsized":"2024-04-18T21:52:15.459239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"32'2","reported_seq":150,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.622406+0000","last_change":"2024-04-18T21:52:12.429008+0000","last_active":"2024-04-18T21:52:13.622406+0000","last_peered":"2024-04-18T21:52:13.622406+0000","last_clean":"2024-04-18T21:52:13.622406+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:13.622406+0000","last_undegraded":"2024-04-18T21:52:13.622406+0000","last_fullsized":"2024-04-18T21:52:13.622406+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:17.873186+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.0017596700000000001,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.416410+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:14.416410+0000","last_peered":"2024-04-18T21:52:14.416410+0000","last_clean":"2024-04-18T21:52:14.416410+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:14.416410+0000","last_undegraded":"2024-04-18T21:52:14.416410+0000","last_fullsized":"2024-04-18T21:52:14.416410+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":144,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.523284+0000","last_change":"2024-04-18T21:52:12.484955+0000","last_active":"2024-04-18T21:52:13.523284+0000","last_peered":"2024-04-18T21:52:13.523284+0000","last_clean":"2024-04-18T21:52:13.523284+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:13.523284+0000","last_undegraded":"2024-04-18T21:52:13.523284+0000","last_fullsized":"2024-04-18T21:52:13.523284+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:11.077616+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.001707351,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.528229+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:14.528229+0000","last_peered":"2024-04-18T21:52:14.528229+0000","last_clean":"2024-04-18T21:52:14.528229+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:14.528229+0000","last_undegraded":"2024-04-18T21:52:14.528229+0000","last_fullsized":"2024-04-18T21:52:14.528229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.456707+0000","last_change":"2024-04-18T21:52:15.458162+0000","last_active":"2024-04-18T21:52:15.456707+0000","last_peered":"2024-04-18T21:52:15.456707+0000","last_clean":"2024-04-18T21:52:15.456707+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:15.456707+0000","last_undegraded":"2024-04-18T21:52:15.456707+0000","last_fullsized":"2024-04-18T21:52:15.456707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.146084+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.0033094909999999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3170,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.621166+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:13.621166+0000","last_peered":"2024-04-18T21:52:13.621166+0000","last_clean":"2024-04-18T21:52:13.621166+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:13.621166+0000","last_undegraded":"2024-04-18T21:52:13.621166+0000","last_fullsized":"2024-04-18T21:52:13.621166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"56'13","reported_seq":163,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.416236+0000","last_change":"2024-04-18T21:52:12.822272+0000","last_active":"2024-04-18T21:52:14.416236+0000","last_peered":"2024-04-18T21:52:14.416236+0000","last_clean":"2024-04-18T21:52:14.416236+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:14.416236+0000","last_undegraded":"2024-04-18T21:52:14.416236+0000","last_fullsized":"2024-04-18T21:52:14.416236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.039867+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.40428434699999999,"stat_sum":{"num_bytes":868352,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.3","version":"59'84","reported_seq":257,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.763076+0000","last_change":"2024-04-18T21:52:15.765341+0000","last_active":"2024-04-18T21:52:15.763076+0000","last_peered":"2024-04-18T21:52:15.763076+0000","last_clean":"2024-04-18T21:52:15.763076+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:15.763076+0000","last_undegraded":"2024-04-18T21:52:15.763076+0000","last_fullsized":"2024-04-18T21:52:15.763076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:24.270053+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.28746011199999999,"stat_sum":{"num_bytes":10403840,"num_objects":11,"num_object_clones":4,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.516016+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:14.516016+0000","last_peered":"2024-04-18T21:52:14.516016+0000","last_clean":"2024-04-18T21:52:14.516016+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:14.516016+0000","last_undegraded":"2024-04-18T21:52:14.516016+0000","last_fullsized":"2024-04-18T21:52:14.516016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"59'22","reported_seq":189,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.957919+0000","last_change":"2024-04-18T21:52:15.959178+0000","last_active":"2024-04-18T21:52:15.957919+0000","last_peered":"2024-04-18T21:52:15.957919+0000","last_clean":"2024-04-18T21:52:15.957919+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:15.957919+0000","last_undegraded":"2024-04-18T21:52:15.957919+0000","last_fullsized":"2024-04-18T21:52:15.957919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.113096+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.50073931699999996,"stat_sum":{"num_bytes":3866624,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.416305+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:14.416305+0000","last_peered":"2024-04-18T21:52:14.416305+0000","last_clean":"2024-04-18T21:52:14.416305+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:14.416305+0000","last_undegraded":"2024-04-18T21:52:14.416305+0000","last_fullsized":"2024-04-18T21:52:14.416305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"51'10","reported_seq":162,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.620046+0000","last_change":"2024-04-18T21:52:12.628380+0000","last_active":"2024-04-18T21:52:13.620046+0000","last_peered":"2024-04-18T21:52:13.620046+0000","last_clean":"2024-04-18T21:52:13.620046+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:13.620046+0000","last_undegraded":"2024-04-18T21:52:13.620046+0000","last_fullsized":"2024-04-18T21:52:13.620046+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:19.798611+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0.102783557,"stat_sum":{"num_bytes":720896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.419935+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:14.419935+0000","last_peered":"2024-04-18T21:52:14.419935+0000","last_clean":"2024-04-18T21:52:14.419935+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:14.419935+0000","last_undegraded":"2024-04-18T21:52:14.419935+0000","last_fullsized":"2024-04-18T21:52:14.419935+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"59'9","reported_seq":169,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.761070+0000","last_change":"2024-04-18T21:52:15.763319+0000","last_active":"2024-04-18T21:52:15.761070+0000","last_peered":"2024-04-18T21:52:15.761070+0000","last_clean":"2024-04-18T21:52:15.761070+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:15.761070+0000","last_undegraded":"2024-04-18T21:52:15.761070+0000","last_fullsized":"2024-04-18T21:52:15.761070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:25.933067+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.30883592399999998,"stat_sum":{"num_bytes":1605632,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.417514+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:14.417514+0000","last_peered":"2024-04-18T21:52:14.417514+0000","last_clean":"2024-04-18T21:52:14.417514+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:14.417514+0000","last_undegraded":"2024-04-18T21:52:14.417514+0000","last_fullsized":"2024-04-18T21:52:14.417514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":261,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.420217+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:14.420217+0000","last_peered":"2024-04-18T21:52:14.420217+0000","last_clean":"2024-04-18T21:52:14.420217+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:14.420217+0000","last_undegraded":"2024-04-18T21:52:14.420217+0000","last_fullsized":"2024-04-18T21:52:14.420217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":149,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.527499+0000","last_change":"2024-04-18T21:52:12.578695+0000","last_active":"2024-04-18T21:52:14.527499+0000","last_peered":"2024-04-18T21:52:14.527499+0000","last_clean":"2024-04-18T21:52:14.527499+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:14.527499+0000","last_undegraded":"2024-04-18T21:52:14.527499+0000","last_fullsized":"2024-04-18T21:52:14.527499+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:08.832807+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.11817699800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.9","version":"29'6","reported_seq":162,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.527731+0000","last_change":"2024-04-18T21:52:12.991095+0000","last_active":"2024-04-18T21:52:14.527731+0000","last_peered":"2024-04-18T21:52:14.527731+0000","last_clean":"2024-04-18T21:52:14.527731+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:14.527731+0000","last_undegraded":"2024-04-18T21:52:14.527731+0000","last_fullsized":"2024-04-18T21:52:14.527731+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:25.867746+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.43105967899999997,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.a","version":"51'13","reported_seq":157,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.560980+0000","last_change":"2024-04-18T21:52:15.562372+0000","last_active":"2024-04-18T21:52:15.560980+0000","last_peered":"2024-04-18T21:52:15.560980+0000","last_clean":"2024-04-18T21:52:15.560980+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:15.560980+0000","last_undegraded":"2024-04-18T21:52:15.560980+0000","last_fullsized":"2024-04-18T21:52:15.560980+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.548343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002066048,"stat_sum":{"num_bytes":1146880,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":3268,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"3.b","version":"56'44","reported_seq":183,"reported_epoch":59,"state":"active+clean+snaptrim","last_fresh":"2024-04-18T21:52:15.551477+0000","last_change":"2024-04-18T21:52:15.551556+0000","last_active":"2024-04-18T21:52:15.551477+0000","last_peered":"2024-04-18T21:52:15.551477+0000","last_clean":"2024-04-18T21:52:15.551477+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:15.551477+0000","last_undegraded":"2024-04-18T21:52:15.551477+0000","last_fullsized":"2024-04-18T21:52:15.551477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.007755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.214296292,"stat_sum":{"num_bytes":4833280,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.c","version":"56'45","reported_seq":179,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.516187+0000","last_change":"2024-04-18T21:52:12.921819+0000","last_active":"2024-04-18T21:52:14.516187+0000","last_peered":"2024-04-18T21:52:14.516187+0000","last_clean":"2024-04-18T21:52:14.516187+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:14.516187+0000","last_undegraded":"2024-04-18T21:52:14.516187+0000","last_fullsized":"2024-04-18T21:52:14.516187+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.809745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50250953700000001,"stat_sum":{"num_bytes":2605056,"num_objects":6,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":22,"num_read_kb":2008,"num_write":57,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":129,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.521419+0000","last_change":"2024-04-18T21:52:12.520506+0000","last_active":"2024-04-18T21:52:14.521419+0000","last_peered":"2024-04-18T21:52:14.521419+0000","last_clean":"2024-04-18T21:52:14.521419+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:14.521419+0000","last_undegraded":"2024-04-18T21:52:14.521419+0000","last_fullsized":"2024-04-18T21:52:14.521419+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:00.576387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101255319,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.e","version":"51'28","reported_seq":166,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.427017+0000","last_change":"2024-04-18T21:52:12.528799+0000","last_active":"2024-04-18T21:52:14.427017+0000","last_peered":"2024-04-18T21:52:14.427017+0000","last_clean":"2024-04-18T21:52:14.427017+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:14.427017+0000","last_undegraded":"2024-04-18T21:52:14.427017+0000","last_fullsized":"2024-04-18T21:52:14.427017+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:46.337113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068866700000000001,"stat_sum":{"num_bytes":2785280,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":4039,"num_write":17,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.f","version":"58'21","reported_seq":156,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.521023+0000","last_change":"2024-04-18T21:52:12.419891+0000","last_active":"2024-04-18T21:52:14.521023+0000","last_peered":"2024-04-18T21:52:14.521023+0000","last_clean":"2024-04-18T21:52:14.521023+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:14.521023+0000","last_undegraded":"2024-04-18T21:52:14.521023+0000","last_fullsized":"2024-04-18T21:52:14.521023+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:11.319109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000957506,"stat_sum":{"num_bytes":4046848,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":2886,"num_write":17,"num_write_kb":4688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.10","version":"59'26","reported_seq":146,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.865053+0000","last_change":"2024-04-18T21:52:15.867407+0000","last_active":"2024-04-18T21:52:15.865053+0000","last_peered":"2024-04-18T21:52:15.865053+0000","last_clean":"2024-04-18T21:52:15.865053+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:15.865053+0000","last_undegraded":"2024-04-18T21:52:15.865053+0000","last_fullsized":"2024-04-18T21:52:15.865053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:18.295879+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0.30822618899999998,"stat_sum":{"num_bytes":3096576,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"3.11","version":"56'28","reported_seq":122,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.526893+0000","last_change":"2024-04-18T21:52:12.990596+0000","last_active":"2024-04-18T21:52:14.526893+0000","last_peered":"2024-04-18T21:52:14.526893+0000","last_clean":"2024-04-18T21:52:14.526893+0000","last_became_active":"2024-04-18T21:52:06.594871+0000","last_became_peered":"2024-04-18T21:52:06.594871+0000","last_unstale":"2024-04-18T21:52:14.526893+0000","last_undegraded":"2024-04-18T21:52:14.526893+0000","last_fullsized":"2024-04-18T21:52:14.526893+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'24","last_scrub_stamp":"2024-04-18T21:52:12.154768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:12.154768+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:29.403318+0000","scrub_duration":1591,"objects_trimmed":0,"snaptrim_duration":0.430775821,"stat_sum":{"num_bytes":3899392,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,10,1,5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.12","version":"56'11","reported_seq":77,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-04-18T21:52:14.658456+0000","last_change":"2024-04-18T21:52:14.260207+0000","last_active":"2024-04-18T21:52:14.658456+0000","last_peered":"2024-04-18T21:52:14.658456+0000","last_clean":"2024-04-18T21:52:14.658456+0000","last_became_active":"2024-04-18T21:52:11.882697+0000","last_became_peered":"2024-04-18T21:52:11.882697+0000","last_unstale":"2024-04-18T21:52:14.658456+0000","last_undegraded":"2024-04-18T21:52:14.658456+0000","last_fullsized":"2024-04-18T21:52:14.658456+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'5","last_scrub_stamp":"2024-04-18T21:52:01.496294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.496294+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21274892400000001,"stat_sum":{"num_bytes":925696,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,10,0,5,6,1],"acting":[9,7,8,10,0,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.13","version":"55'35","reported_seq":90,"reported_epoch":57,"state":"peering","last_fresh":"2024-04-18T21:52:14.156366+0000","last_change":"2024-04-18T21:52:14.421635+0000","last_active":"2024-04-18T21:52:14.421635+0000","last_peered":"2024-04-18T21:52:14.156366+0000","last_clean":"2024-04-18T21:52:07.879241+0000","last_became_active":"2024-04-18T21:52:11.015356+0000","last_became_peered":"2024-04-18T21:52:11.015356+0000","last_unstale":"2024-04-18T21:52:14.156366+0000","last_undegraded":"2024-04-18T21:52:14.156366+0000","last_fullsized":"2024-04-18T21:52:10.365351+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.207004411,"stat_sum":{"num_bytes":3563520,"num_objects":8,"num_object_clones":2,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":3175,"num_write":19,"num_write_kb":4376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4,8,6,11,1,9],"acting":[7,5,4,8,6,11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2,3,4,5,6,8,9,11],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":51200000,"num_objects":71,"num_object_clones":21,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":3,"num_read":127,"num_read_kb":21343,"num_write":161,"num_write_kb":31312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2539520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99385344,"data_stored":114380800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":471,"ondisk_log_size":471,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378637,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38472,"kb_used_data":7288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333368,"statfs":{"total":96636764160,"available":96597368832,"internally_reserved":0,"allocated":7462912,"data_stored":8772460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378637,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41668,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330172,"statfs":{"total":96636764160,"available":96594096128,"internally_reserved":0,"allocated":10739712,"data_stored":12167787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378638,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":10188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":10432512,"data_stored":11512684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":12172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":12464128,"data_stored":13609836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378637,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39108,"kb_used_data":7928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332732,"statfs":{"total":96636764160,"available":96596717568,"internally_reserved":0,"allocated":8118272,"data_stored":8759915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411340,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":5236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":5361664,"data_stored":5224861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411340,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35076,"kb_used_data":8056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336764,"statfs":{"total":96636764160,"available":96600846336,"internally_reserved":0,"allocated":8249344,"data_stored":9858187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411340,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41856,"kb_used_data":10672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329984,"statfs":{"total":96636764160,"available":96593903616,"internally_reserved":0,"allocated":10928128,"data_stored":12442219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":10060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":10301440,"data_stored":11053469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34520,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337320,"statfs":{"total":96636764160,"available":96601415680,"internally_reserved":0,"allocated":7671808,"data_stored":7592893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444044,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34040,"kb_used_data":7012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337800,"statfs":{"total":96636764160,"available":96601907200,"internally_reserved":0,"allocated":7180288,"data_stored":7055773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444044,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":6072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":6217728,"data_stored":6371741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6852608,"data_stored":8052736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11800576,"data_stored":13225984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10326016,"data_stored":12406784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8421376,"data_stored":10006528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9768960,"data_stored":11128832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10276864,"data_stored":12066816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":10420224,"data_stored":11788288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7065600,"data_stored":8953856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5984256,"data_stored":6410240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4722688,"data_stored":4857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6799360,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6946816,"data_stored":7094272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:16.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-18T21:52:16.397 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:16.395+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:16.405 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:16.405 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:16.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is set 2024-04-18T21:52:16.449 INFO:tasks.rados.rados.0.smithi097.stdout:167: done (6 left) 2024-04-18T21:52:16.449 INFO:tasks.rados.rados.0.smithi097.stdout:168: done (5 left) 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:170: done (4 left) 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:171: done (3 left) 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:173: done (2 left) 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:176: done (1 left) 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:178: done (0 left) 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:179: copy_from oid 17 from oid 8 current snap is 14 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:180: read oid 32 snap -1 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:180: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:52:16.450 INFO:tasks.rados.rados.0.smithi097.stdout:181: snap_remove snap 10 2024-04-18T21:52:16.481 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:16.481+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:16.563 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:16.563 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:16.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:16.582 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:16.579+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:16.683 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:16.679+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:16.693 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:16.694 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:16.701 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:16.701 INFO:teuthology.orchestra.run.smithi097.stdout:{"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":59,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":60,"num_osds":12,"num_up_osds":12,"osd_up_since":1713477089,"num_in_osds":12,"osd_in_since":1713477081,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":32},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"peering","count":1},{"state_name":"active+clean+snaptrim","count":1}],"num_pgs":35,"num_pools":3,"num_objects":73,"data_bytes":51790368,"bytes_used":462766080,"bytes_avail":1159178403840,"bytes_total":1159641169920,"inactive_pgs_ratio":0.02857142873108387,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":249786,"recovering_keys_per_sec":0,"num_objects_recovered":4,"num_bytes_recovered":999424,"num_keys_recovered":0,"read_bytes_sec":1868533,"read_op_per_sec":4,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-18T21:51:17.632974+0000","services":{}},"progress_events":{}} 2024-04-18T21:52:16.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:16.721 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-04-18T21:52:16.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:16.783 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:16.779+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:16.836 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:16.836 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:16.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:16.884 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:16.885+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:16.985 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:16.985+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:17.008 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:17.008 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:17.023 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:17.086 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:17.083+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:17.101 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:17.102 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:17.118 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-04-18T21:52:16.444823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51790368,"num_objects":73,"num_object_clones":21,"num_object_copies":572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":3,"num_read":253,"num_read_kb":21452,"num_write":342,"num_write_kb":34250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2539520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":239,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":451920,"kb_used_data":102664,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1132010160,"statfs":{"total":1159641169920,"available":1159178403840,"internally_reserved":0,"allocated":105127936,"data_stored":114421825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":357414205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,1,1,2,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10018816,"num_objects":11,"num_object_clones":10,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":-6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":19,"num_read_kb":7301,"num_write":-10,"num_write_kb":5816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.037955"},"pg_stats":[{"pgid":"3.19","version":"56'22","reported_seq":181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.627044+0000","last_change":"2024-04-18T21:52:12.990105+0000","last_active":"2024-04-18T21:52:14.627044+0000","last_peered":"2024-04-18T21:52:14.627044+0000","last_clean":"2024-04-18T21:52:14.627044+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:14.627044+0000","last_undegraded":"2024-04-18T21:52:14.627044+0000","last_fullsized":"2024-04-18T21:52:14.627044+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.552392+0000","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0.43035931799999999,"stat_sum":{"num_bytes":1376256,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.18","version":"51'20","reported_seq":174,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.526784+0000","last_change":"2024-04-18T21:52:12.561250+0000","last_active":"2024-04-18T21:52:14.526784+0000","last_peered":"2024-04-18T21:52:14.526784+0000","last_clean":"2024-04-18T21:52:14.526784+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:14.526784+0000","last_undegraded":"2024-04-18T21:52:14.526784+0000","last_fullsized":"2024-04-18T21:52:14.526784+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:07.165130+0000","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0.10117693699999999,"stat_sum":{"num_bytes":2523136,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2258,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.17","version":"51'8","reported_seq":155,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.623152+0000","last_change":"2024-04-18T21:52:12.486846+0000","last_active":"2024-04-18T21:52:13.623152+0000","last_peered":"2024-04-18T21:52:13.623152+0000","last_clean":"2024-04-18T21:52:13.623152+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:13.623152+0000","last_undegraded":"2024-04-18T21:52:13.623152+0000","last_fullsized":"2024-04-18T21:52:13.623152+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:19.410147+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.0026122879999999999,"stat_sum":{"num_bytes":671744,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.16","version":"32'3","reported_seq":154,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.521746+0000","last_change":"2024-04-18T21:52:12.428844+0000","last_active":"2024-04-18T21:52:13.521746+0000","last_peered":"2024-04-18T21:52:13.521746+0000","last_clean":"2024-04-18T21:52:13.521746+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:13.521746+0000","last_undegraded":"2024-04-18T21:52:13.521746+0000","last_fullsized":"2024-04-18T21:52:13.521746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:05.631013+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.00193593,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.15","version":"51'9","reported_seq":160,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.417338+0000","last_change":"2024-04-18T21:52:12.822985+0000","last_active":"2024-04-18T21:52:14.417338+0000","last_peered":"2024-04-18T21:52:14.417338+0000","last_clean":"2024-04-18T21:52:14.417338+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:14.417338+0000","last_undegraded":"2024-04-18T21:52:14.417338+0000","last_fullsized":"2024-04-18T21:52:14.417338+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:56.919539+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.30495551599999998,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.14","version":"51'8","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.455291+0000","last_change":"2024-04-18T21:52:15.456411+0000","last_active":"2024-04-18T21:52:15.455291+0000","last_peered":"2024-04-18T21:52:15.455291+0000","last_clean":"2024-04-18T21:52:15.455291+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:15.455291+0000","last_undegraded":"2024-04-18T21:52:15.455291+0000","last_fullsized":"2024-04-18T21:52:15.455291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.605862+0000","scrub_duration":1515,"objects_trimmed":0,"snaptrim_duration":0.0017272139999999999,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.459239+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:15.459239+0000","last_peered":"2024-04-18T21:52:15.459239+0000","last_clean":"2024-04-18T21:52:15.459239+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:15.459239+0000","last_undegraded":"2024-04-18T21:52:15.459239+0000","last_fullsized":"2024-04-18T21:52:15.459239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"32'2","reported_seq":150,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.622406+0000","last_change":"2024-04-18T21:52:12.429008+0000","last_active":"2024-04-18T21:52:13.622406+0000","last_peered":"2024-04-18T21:52:13.622406+0000","last_clean":"2024-04-18T21:52:13.622406+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:13.622406+0000","last_undegraded":"2024-04-18T21:52:13.622406+0000","last_fullsized":"2024-04-18T21:52:13.622406+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:17.873186+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.0017596700000000001,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.416410+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:14.416410+0000","last_peered":"2024-04-18T21:52:14.416410+0000","last_clean":"2024-04-18T21:52:14.416410+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:14.416410+0000","last_undegraded":"2024-04-18T21:52:14.416410+0000","last_fullsized":"2024-04-18T21:52:14.416410+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":144,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.523284+0000","last_change":"2024-04-18T21:52:12.484955+0000","last_active":"2024-04-18T21:52:13.523284+0000","last_peered":"2024-04-18T21:52:13.523284+0000","last_clean":"2024-04-18T21:52:13.523284+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:13.523284+0000","last_undegraded":"2024-04-18T21:52:13.523284+0000","last_fullsized":"2024-04-18T21:52:13.523284+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:11.077616+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.001707351,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.528229+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:14.528229+0000","last_peered":"2024-04-18T21:52:14.528229+0000","last_clean":"2024-04-18T21:52:14.528229+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:14.528229+0000","last_undegraded":"2024-04-18T21:52:14.528229+0000","last_fullsized":"2024-04-18T21:52:14.528229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.456707+0000","last_change":"2024-04-18T21:52:15.458162+0000","last_active":"2024-04-18T21:52:15.456707+0000","last_peered":"2024-04-18T21:52:15.456707+0000","last_clean":"2024-04-18T21:52:15.456707+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:15.456707+0000","last_undegraded":"2024-04-18T21:52:15.456707+0000","last_fullsized":"2024-04-18T21:52:15.456707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.146084+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.0033094909999999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3170,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.621166+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:13.621166+0000","last_peered":"2024-04-18T21:52:13.621166+0000","last_clean":"2024-04-18T21:52:13.621166+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:13.621166+0000","last_undegraded":"2024-04-18T21:52:13.621166+0000","last_fullsized":"2024-04-18T21:52:13.621166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"56'13","reported_seq":163,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.416236+0000","last_change":"2024-04-18T21:52:12.822272+0000","last_active":"2024-04-18T21:52:14.416236+0000","last_peered":"2024-04-18T21:52:14.416236+0000","last_clean":"2024-04-18T21:52:14.416236+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:14.416236+0000","last_undegraded":"2024-04-18T21:52:14.416236+0000","last_fullsized":"2024-04-18T21:52:14.416236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.039867+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.40428434699999999,"stat_sum":{"num_bytes":868352,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.3","version":"59'84","reported_seq":257,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.763076+0000","last_change":"2024-04-18T21:52:15.765341+0000","last_active":"2024-04-18T21:52:15.763076+0000","last_peered":"2024-04-18T21:52:15.763076+0000","last_clean":"2024-04-18T21:52:15.763076+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:15.763076+0000","last_undegraded":"2024-04-18T21:52:15.763076+0000","last_fullsized":"2024-04-18T21:52:15.763076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:24.270053+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.28746011199999999,"stat_sum":{"num_bytes":10403840,"num_objects":11,"num_object_clones":4,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.516016+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:14.516016+0000","last_peered":"2024-04-18T21:52:14.516016+0000","last_clean":"2024-04-18T21:52:14.516016+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:14.516016+0000","last_undegraded":"2024-04-18T21:52:14.516016+0000","last_fullsized":"2024-04-18T21:52:14.516016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"59'22","reported_seq":189,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.957919+0000","last_change":"2024-04-18T21:52:15.959178+0000","last_active":"2024-04-18T21:52:15.957919+0000","last_peered":"2024-04-18T21:52:15.957919+0000","last_clean":"2024-04-18T21:52:15.957919+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:15.957919+0000","last_undegraded":"2024-04-18T21:52:15.957919+0000","last_fullsized":"2024-04-18T21:52:15.957919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.113096+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.50073931699999996,"stat_sum":{"num_bytes":3866624,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.416305+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:14.416305+0000","last_peered":"2024-04-18T21:52:14.416305+0000","last_clean":"2024-04-18T21:52:14.416305+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:14.416305+0000","last_undegraded":"2024-04-18T21:52:14.416305+0000","last_fullsized":"2024-04-18T21:52:14.416305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"51'10","reported_seq":162,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-18T21:52:13.620046+0000","last_change":"2024-04-18T21:52:12.628380+0000","last_active":"2024-04-18T21:52:13.620046+0000","last_peered":"2024-04-18T21:52:13.620046+0000","last_clean":"2024-04-18T21:52:13.620046+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:13.620046+0000","last_undegraded":"2024-04-18T21:52:13.620046+0000","last_fullsized":"2024-04-18T21:52:13.620046+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:19.798611+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0.102783557,"stat_sum":{"num_bytes":720896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.419935+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:14.419935+0000","last_peered":"2024-04-18T21:52:14.419935+0000","last_clean":"2024-04-18T21:52:14.419935+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:14.419935+0000","last_undegraded":"2024-04-18T21:52:14.419935+0000","last_fullsized":"2024-04-18T21:52:14.419935+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"59'9","reported_seq":169,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.761070+0000","last_change":"2024-04-18T21:52:15.763319+0000","last_active":"2024-04-18T21:52:15.761070+0000","last_peered":"2024-04-18T21:52:15.761070+0000","last_clean":"2024-04-18T21:52:15.761070+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:15.761070+0000","last_undegraded":"2024-04-18T21:52:15.761070+0000","last_fullsized":"2024-04-18T21:52:15.761070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:25.933067+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.30883592399999998,"stat_sum":{"num_bytes":1605632,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.417514+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:14.417514+0000","last_peered":"2024-04-18T21:52:14.417514+0000","last_clean":"2024-04-18T21:52:14.417514+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:14.417514+0000","last_undegraded":"2024-04-18T21:52:14.417514+0000","last_fullsized":"2024-04-18T21:52:14.417514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":261,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.420217+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:14.420217+0000","last_peered":"2024-04-18T21:52:14.420217+0000","last_clean":"2024-04-18T21:52:14.420217+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:14.420217+0000","last_undegraded":"2024-04-18T21:52:14.420217+0000","last_fullsized":"2024-04-18T21:52:14.420217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":149,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.527499+0000","last_change":"2024-04-18T21:52:12.578695+0000","last_active":"2024-04-18T21:52:14.527499+0000","last_peered":"2024-04-18T21:52:14.527499+0000","last_clean":"2024-04-18T21:52:14.527499+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:14.527499+0000","last_undegraded":"2024-04-18T21:52:14.527499+0000","last_fullsized":"2024-04-18T21:52:14.527499+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:08.832807+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.11817699800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.9","version":"29'6","reported_seq":162,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.527731+0000","last_change":"2024-04-18T21:52:12.991095+0000","last_active":"2024-04-18T21:52:14.527731+0000","last_peered":"2024-04-18T21:52:14.527731+0000","last_clean":"2024-04-18T21:52:14.527731+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:14.527731+0000","last_undegraded":"2024-04-18T21:52:14.527731+0000","last_fullsized":"2024-04-18T21:52:14.527731+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:25.867746+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.43105967899999997,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.a","version":"51'13","reported_seq":157,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.560980+0000","last_change":"2024-04-18T21:52:15.562372+0000","last_active":"2024-04-18T21:52:15.560980+0000","last_peered":"2024-04-18T21:52:15.560980+0000","last_clean":"2024-04-18T21:52:15.560980+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:15.560980+0000","last_undegraded":"2024-04-18T21:52:15.560980+0000","last_fullsized":"2024-04-18T21:52:15.560980+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.548343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002066048,"stat_sum":{"num_bytes":1146880,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":3268,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"3.b","version":"56'44","reported_seq":183,"reported_epoch":59,"state":"active+clean+snaptrim","last_fresh":"2024-04-18T21:52:15.551477+0000","last_change":"2024-04-18T21:52:15.551556+0000","last_active":"2024-04-18T21:52:15.551477+0000","last_peered":"2024-04-18T21:52:15.551477+0000","last_clean":"2024-04-18T21:52:15.551477+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:15.551477+0000","last_undegraded":"2024-04-18T21:52:15.551477+0000","last_fullsized":"2024-04-18T21:52:15.551477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.007755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.214296292,"stat_sum":{"num_bytes":4833280,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.c","version":"56'45","reported_seq":179,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.516187+0000","last_change":"2024-04-18T21:52:12.921819+0000","last_active":"2024-04-18T21:52:14.516187+0000","last_peered":"2024-04-18T21:52:14.516187+0000","last_clean":"2024-04-18T21:52:14.516187+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:14.516187+0000","last_undegraded":"2024-04-18T21:52:14.516187+0000","last_fullsized":"2024-04-18T21:52:14.516187+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:10.809745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50250953700000001,"stat_sum":{"num_bytes":2605056,"num_objects":6,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":22,"num_read_kb":2008,"num_write":57,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":129,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.521419+0000","last_change":"2024-04-18T21:52:12.520506+0000","last_active":"2024-04-18T21:52:14.521419+0000","last_peered":"2024-04-18T21:52:14.521419+0000","last_clean":"2024-04-18T21:52:14.521419+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:14.521419+0000","last_undegraded":"2024-04-18T21:52:14.521419+0000","last_fullsized":"2024-04-18T21:52:14.521419+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:00.576387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101255319,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.e","version":"51'28","reported_seq":166,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.427017+0000","last_change":"2024-04-18T21:52:12.528799+0000","last_active":"2024-04-18T21:52:14.427017+0000","last_peered":"2024-04-18T21:52:14.427017+0000","last_clean":"2024-04-18T21:52:14.427017+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:14.427017+0000","last_undegraded":"2024-04-18T21:52:14.427017+0000","last_fullsized":"2024-04-18T21:52:14.427017+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:46.337113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068866700000000001,"stat_sum":{"num_bytes":2785280,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":4039,"num_write":17,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.f","version":"58'21","reported_seq":156,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.521023+0000","last_change":"2024-04-18T21:52:12.419891+0000","last_active":"2024-04-18T21:52:14.521023+0000","last_peered":"2024-04-18T21:52:14.521023+0000","last_clean":"2024-04-18T21:52:14.521023+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:14.521023+0000","last_undegraded":"2024-04-18T21:52:14.521023+0000","last_fullsized":"2024-04-18T21:52:14.521023+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:11.319109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000957506,"stat_sum":{"num_bytes":4046848,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":2886,"num_write":17,"num_write_kb":4688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.10","version":"59'26","reported_seq":146,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.865053+0000","last_change":"2024-04-18T21:52:15.867407+0000","last_active":"2024-04-18T21:52:15.865053+0000","last_peered":"2024-04-18T21:52:15.865053+0000","last_clean":"2024-04-18T21:52:15.865053+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:15.865053+0000","last_undegraded":"2024-04-18T21:52:15.865053+0000","last_fullsized":"2024-04-18T21:52:15.865053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:18.295879+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0.30822618899999998,"stat_sum":{"num_bytes":3096576,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"3.11","version":"56'28","reported_seq":122,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-18T21:52:14.526893+0000","last_change":"2024-04-18T21:52:12.990596+0000","last_active":"2024-04-18T21:52:14.526893+0000","last_peered":"2024-04-18T21:52:14.526893+0000","last_clean":"2024-04-18T21:52:14.526893+0000","last_became_active":"2024-04-18T21:52:06.594871+0000","last_became_peered":"2024-04-18T21:52:06.594871+0000","last_unstale":"2024-04-18T21:52:14.526893+0000","last_undegraded":"2024-04-18T21:52:14.526893+0000","last_fullsized":"2024-04-18T21:52:14.526893+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'24","last_scrub_stamp":"2024-04-18T21:52:12.154768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:12.154768+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:29.403318+0000","scrub_duration":1591,"objects_trimmed":0,"snaptrim_duration":0.430775821,"stat_sum":{"num_bytes":3899392,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,10,1,5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.12","version":"56'11","reported_seq":77,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-04-18T21:52:14.658456+0000","last_change":"2024-04-18T21:52:14.260207+0000","last_active":"2024-04-18T21:52:14.658456+0000","last_peered":"2024-04-18T21:52:14.658456+0000","last_clean":"2024-04-18T21:52:14.658456+0000","last_became_active":"2024-04-18T21:52:11.882697+0000","last_became_peered":"2024-04-18T21:52:11.882697+0000","last_unstale":"2024-04-18T21:52:14.658456+0000","last_undegraded":"2024-04-18T21:52:14.658456+0000","last_fullsized":"2024-04-18T21:52:14.658456+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"33'5","last_scrub_stamp":"2024-04-18T21:52:01.496294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.496294+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21274892400000001,"stat_sum":{"num_bytes":925696,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,10,0,5,6,1],"acting":[9,7,8,10,0,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.13","version":"55'35","reported_seq":90,"reported_epoch":57,"state":"peering","last_fresh":"2024-04-18T21:52:14.156366+0000","last_change":"2024-04-18T21:52:14.421635+0000","last_active":"2024-04-18T21:52:14.421635+0000","last_peered":"2024-04-18T21:52:14.156366+0000","last_clean":"2024-04-18T21:52:07.879241+0000","last_became_active":"2024-04-18T21:52:11.015356+0000","last_became_peered":"2024-04-18T21:52:11.015356+0000","last_unstale":"2024-04-18T21:52:14.156366+0000","last_undegraded":"2024-04-18T21:52:14.156366+0000","last_fullsized":"2024-04-18T21:52:10.365351+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.207004411,"stat_sum":{"num_bytes":3563520,"num_objects":8,"num_object_clones":2,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":3175,"num_write":19,"num_write_kb":4376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4,8,6,11,1,9],"acting":[7,5,4,8,6,11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2,3,4,5,6,8,9,11],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":51200000,"num_objects":71,"num_object_clones":21,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":3,"num_read":127,"num_read_kb":21343,"num_write":161,"num_write_kb":31312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2539520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99385344,"data_stored":114380800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":471,"ondisk_log_size":471,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378637,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38472,"kb_used_data":7288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333368,"statfs":{"total":96636764160,"available":96597368832,"internally_reserved":0,"allocated":7462912,"data_stored":8772460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378637,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41668,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330172,"statfs":{"total":96636764160,"available":96594096128,"internally_reserved":0,"allocated":10739712,"data_stored":12167787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378638,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":10188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":10432512,"data_stored":11512684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":12172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":12464128,"data_stored":13609836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378637,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39108,"kb_used_data":7928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332732,"statfs":{"total":96636764160,"available":96596717568,"internally_reserved":0,"allocated":8118272,"data_stored":8759915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411340,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":5236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":5361664,"data_stored":5224861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411340,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35076,"kb_used_data":8056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336764,"statfs":{"total":96636764160,"available":96600846336,"internally_reserved":0,"allocated":8249344,"data_stored":9858187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411340,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41856,"kb_used_data":10672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329984,"statfs":{"total":96636764160,"available":96593903616,"internally_reserved":0,"allocated":10928128,"data_stored":12442219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":10060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":10301440,"data_stored":11053469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411339,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34520,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337320,"statfs":{"total":96636764160,"available":96601415680,"internally_reserved":0,"allocated":7671808,"data_stored":7592893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444044,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34040,"kb_used_data":7012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337800,"statfs":{"total":96636764160,"available":96601907200,"internally_reserved":0,"allocated":7180288,"data_stored":7055773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444044,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":6072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":6217728,"data_stored":6371741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6852608,"data_stored":8052736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11800576,"data_stored":13225984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10326016,"data_stored":12406784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8421376,"data_stored":10006528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9768960,"data_stored":11128832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10276864,"data_stored":12066816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":10420224,"data_stored":11788288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7065600,"data_stored":8953856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5984256,"data_stored":6410240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4722688,"data_stored":4857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6799360,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6946816,"data_stored":7094272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:17.154 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:17.155 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:17.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:17.187 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:17.183+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:17.288 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:17.287+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:17.337 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:17.337 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:17.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:17.362 INFO:tasks.rados.rados.0.smithi097.stdout:181: done (2 left) 2024-04-18T21:52:17.363 INFO:tasks.rados.rados.0.smithi097.stdout:182: rmattr oid 13 current snap is 14 2024-04-18T21:52:17.363 INFO:tasks.rados.rados.0.smithi097.stdout:182: done (2 left) 2024-04-18T21:52:17.363 INFO:tasks.rados.rados.0.smithi097.stdout:183: copy_from oid 27 from oid 29 current snap is 14 2024-04-18T21:52:17.363 INFO:tasks.rados.rados.0.smithi097.stdout:179: finishing copy_from to smithi09731216-17 2024-04-18T21:52:17.363 INFO:tasks.rados.rados.0.smithi097.stdout:179: got expected ENOENT (src dne) 2024-04-18T21:52:17.363 INFO:tasks.rados.rados.0.smithi097.stdout:179: finishing copy_from racing read to smithi09731216-17 2024-04-18T21:52:17.363 INFO:tasks.rados.rados.0.smithi097.stdout:180: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:52:17.389 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:17.387+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:17.403 INFO:tasks.rados.rados.0.smithi097.stdout:179: done (2 left) 2024-04-18T21:52:17.403 INFO:tasks.rados.rados.0.smithi097.stdout:180: done (1 left) 2024-04-18T21:52:17.403 INFO:tasks.rados.rados.0.smithi097.stdout:184: snap_remove snap 12 2024-04-18T21:52:17.491 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:17.487+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:17.518 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:17.518 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:17.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:17.591 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:17.593+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:17.656 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:17.657 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:17.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:17.721 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:17.719+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:17.793 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:17.791+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:17.793 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:17.794 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:17.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:17.894 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:17.893+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:17.977 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:17.977 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:17.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:17.995 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:17.993+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:18.095 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:18.091+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:18.114 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:18.115 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:18.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:18.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:18.196 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:18.191+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:18.250 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:18.251 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:18.267 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:18.281 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:18.281 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:18.298 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:18.297+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:18.298 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:18.299 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:18.315 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:18.315 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:18.331 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:18.332 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:18.349 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:18.349 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:18.360 INFO:tasks.rados.rados.0.smithi097.stdout:184: done (1 left) 2024-04-18T21:52:18.360 INFO:tasks.rados.rados.0.smithi097.stdout:185: read oid 3 snap -1 2024-04-18T21:52:18.360 INFO:tasks.rados.rados.0.smithi097.stdout:185: expect (ObjNum 66 snap 4 seq_num 66) 2024-04-18T21:52:18.360 INFO:tasks.rados.rados.0.smithi097.stdout:186: rollback oid 36 current snap is 14 2024-04-18T21:52:18.360 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 36 to 7 2024-04-18T21:52:18.360 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 45 current snap is 14 2024-04-18T21:52:18.360 INFO:tasks.rados.rados.0.smithi097.stdout:187: seq_num 84 ranges {0=655360,655360=638976,1294336=131072} 2024-04-18T21:52:18.365 INFO:tasks.rados.rados.0.smithi097.stdout:187: writing smithi09731216-45 from 0 to 655360 tid 1 2024-04-18T21:52:18.371 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:18.372 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:18.373 INFO:tasks.rados.rados.0.smithi097.stdout:187: writing smithi09731216-45 from 655360 to 1294336 tid 2 2024-04-18T21:52:18.374 INFO:tasks.rados.rados.0.smithi097.stdout:187: writing smithi09731216-45 from 1294336 to 1425408 tid 3 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:188: read oid 43 snap -1 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:188: expect (ObjNum 74 snap 8 seq_num 74) 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:189: rollback oid 48 current snap is 14 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 48 to 8 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:183: finishing copy_from to smithi09731216-27 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:183: got expected ENOENT (src dne) 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:183: finishing copy_from racing read to smithi09731216-27 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:183: done (5 left) 2024-04-18T21:52:18.375 INFO:tasks.rados.rados.0.smithi097.stdout:190: delete oid 25 current snap is 14 2024-04-18T21:52:18.390 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:18.391 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:18.397 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:18.395+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:18.409 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:18.409 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:18.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:18.409 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:18.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.409 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 36/26 ECSubWrite(tid=36, reqid=client.4509.0:216, at_version=36'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:52:18.409 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:18.409 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.387804763999998, 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30205913200000001, 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.35e-07 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018570+0000", 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.438e-05 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018583+0000", 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2928999999999999e-05 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118702+0000", 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1001194 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218952+0000", 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10025022 2024-04-18T21:52:18.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218966+0000", 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3709e-05 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320525+0000", 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10155911099999999 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320604+0000", 2024-04-18T21:52:18.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9147999999999995e-05 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:239 3.as0 3:5bb4af71:::smithi09731216-7:head [delete] snapc 4=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.392317833, 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40857630099999998, 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:18.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014034+0000", 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2500000000000001e-06 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014044+0000", 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1889999999999996e-06 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014067+0000", 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3251000000000002e-05 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114145+0000", 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078193 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114306+0000", 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161044 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114320+0000", 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4044e-05 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317101+0000", 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20278075300000001 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317362+0000", 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026128799999999998 2024-04-18T21:52:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.319467+0000", 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021048719999999998 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.326280+0000", 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006813375 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422400+0000", 2024-04-18T21:52:18.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096119880000000005 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422608+0000", 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020816199999999999 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:308 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~737280] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.315877756999999, 2024-04-18T21:52:18.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69365002799999997, 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 308 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090475+0000", 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.976e-06 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091768+0000", 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00129317 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091803+0000", 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4502999999999998e-05 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091826+0000", 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3235999999999999e-05 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270501+0000", 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.178674888 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270680+0000", 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017917599999999999 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.273556+0000", 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028757610000000001 2024-04-18T21:52:18.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281580+0000", 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080240799999999994 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783886+0000", 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50230573000000001 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784122+0000", 2024-04-18T21:52:18.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023650799999999999 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:309 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.313796763999999, 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69216249600000002, 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.426 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 309 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092555+0000", 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093122+0000", 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056744499999999997 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093146+0000", 2024-04-18T21:52:18.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128999999999999e-05 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093165+0000", 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8300000000000001e-05 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275352+0000", 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18218714899999999 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275632+0000", 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028059100000000001 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.277549+0000", 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001917087 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281584+0000", 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004034513 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784502+0000", 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50291762600000001 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784716+0000", 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021415599999999999 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:310 3.as0 3:5bb4af71:::smithi09731216-7:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:18.431 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.313155594000001, 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69187687499999995, 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 310 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093196+0000", 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093210+0000", 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3517e-05 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093227+0000", 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7050000000000001e-05 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093243+0000", 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5965999999999999e-05 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279008+0000", 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.185765398 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279287+0000", 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027941400000000002 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.280700+0000", 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412581 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482172+0000", 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20147153900000001 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784956+0000", 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30278430299999998 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785071+0000", 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115454 2024-04-18T21:52:18.436 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:311 3.as0 3:5bb4af71:::smithi09731216-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.313080641999999, 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99553992999999996, 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.437 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 311 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093271+0000", 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093278+0000", 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1459999999999999e-06 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093295+0000", 2024-04-18T21:52:18.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7408999999999999e-05 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093328+0000", 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2276000000000001e-05 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785901+0000", 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.692573628 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.786181+0000", 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027934999999999999 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.088809+0000", 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30262875700000003 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=14, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.370158+0000", 2024-04-18T21:52:18.441 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.036192238999998, 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21261686199999999, 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370158+0000", 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370158+0000", 2024-04-18T21:52:18.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370159+0000", 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3599999999999999e-06 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370167+0000", 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0430000000000008e-06 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370188+0000", 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0599999999999999e-05 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370202+0000", 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4392999999999999e-05 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582740+0000", 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.212537737 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582775+0000", 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4728999999999999e-05 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.370223+0000", 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.036127196999999, 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21269959699999999, 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.445 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370223+0000", 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370223+0000", 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370223+0000", 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7000000000000005e-07 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370227+0000", 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.433e-06 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370237+0000", 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.370244+0000", 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5399999999999998e-06 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582886+0000", 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.212642205 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.582922+0000", 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6099000000000001e-05 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.448 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.026261187999999, 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30330003799999999, 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380090+0000", 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.47e-07 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380096+0000", 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6100000000000002e-06 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380115+0000", 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8986000000000001e-05 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380129+0000", 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4229e-05 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683350+0000", 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30322019500000003 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683389+0000", 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9171000000000002e-05 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=14, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:18.452 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.025018654, 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30209698499999998, 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381333+0000", 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.843e-06 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381343+0000", 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8700000000000004e-06 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381368+0000", 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4613e-05 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381381+0000", 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2857000000000001e-05 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683415+0000", 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30203418999999998 2024-04-18T21:52:18.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683428+0000", 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3611999999999999e-05 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.382104+0000", 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.024246036000001, 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.301352913, 2024-04-18T21:52:18.456 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382104+0000", 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382104+0000", 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382105+0000", 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.443e-06 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382114+0000", 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2130000000000008e-06 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.382133+0000", 2024-04-18T21:52:18.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9830000000000002e-05 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482231+0000", 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10009760600000001 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683445+0000", 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20121439399999999 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683457+0000", 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1426999999999999e-05 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=13, last_complete=41'9, committed=1, applied=1))", 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.923785744, 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30158400099999999, 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.460 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482566+0000", 2024-04-18T21:52:18.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4360000000000001e-06 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482574+0000", 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4700000000000002e-06 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482593+0000", 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9015999999999999e-05 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482608+0000", 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5023e-05 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783615+0000", 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30100726999999999 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784148+0000", 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053278600000000005 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.463 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=14, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.923697373, 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30208896099999999, 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5000000000000002e-07 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482657+0000", 2024-04-18T21:52:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1269999999999996e-06 2024-04-18T21:52:18.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.466 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:18.467 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:18.467 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482672+0000", 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4688999999999999e-05 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482683+0000", 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1165999999999999e-05 2024-04-18T21:52:18.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784263+0000", 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30157952599999999 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784742+0000", 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047880299999999998 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.923653041000001, 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30306911800000003, 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482701+0000", 2024-04-18T21:52:18.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1470000000000001e-06 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482712+0000", 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.182e-05 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482719+0000", 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9129999999999997e-06 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784839+0000", 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30211942400000003 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785766+0000", 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092735400000000003 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.4.19:4 3.as0 3:561bcae7:::smithi09731216-2:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:59.327830+0000", 2024-04-18T21:52:18.473 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.078519923000002, 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.24013599099999999, 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327830+0000", 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327830+0000", 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327832+0000", 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0480000000000001e-06 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327839+0000", 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9600000000000003e-06 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327851+0000", 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2317000000000001e-05 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327861+0000", 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1689999999999998e-06 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.327941+0000", 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9963000000000007e-05 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.328016+0000", 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5492000000000002e-05 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.567966+0000", 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.239950042 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.477 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:418 3.as0 3:5bb4af71:::smithi09731216-7:a [read 0~1114112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:13.414679+0000", 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.9916713169999998, 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21525735800000001, 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 418 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.414679+0000", 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.414679+0000", 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.414680+0000", 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2899999999999999e-06 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.414692+0000", 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.172e-05 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.414711+0000", 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8992999999999999e-05 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.414725+0000", 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4283e-05 2024-04-18T21:52:18.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.415151+0000", 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042613999999999998 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.415168+0000", 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7136e-05 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.525175+0000", 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11000686899999999 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.525405+0000", 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022997 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:13.629936+0000", 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10453095699999999 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.482 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:427 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.9609129169999999, 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38249087900000001, 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.483 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 427 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445440+0000", 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2930000000000001e-06 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445454+0000", 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3883000000000001e-05 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445492+0000", 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8090000000000003e-05 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445519+0000", 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6452999999999999e-05 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445801+0000", 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028185900000000003 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445822+0000", 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1418999999999999e-05 2024-04-18T21:52:18.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.550566+0000", 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104743788 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.550788+0000", 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022258600000000001 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.552194+0000", 2024-04-18T21:52:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001405688 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659810+0000", 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107616082 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.827786+0000", 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16797619999999999 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.827928+0000", 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000141538 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:428 3.10s0 3:0e754656:::smithi09731216-19:head [rollback 9] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.960775903, 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38329886600000002, 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.489 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 428 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445576+0000", 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.514e-06 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445583+0000", 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6429999999999995e-06 2024-04-18T21:52:18.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445626+0000", 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2778999999999998e-05 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445644+0000", 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7597000000000001e-05 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445947+0000", 2024-04-18T21:52:18.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000303462 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445957+0000", 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6059999999999992e-06 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554327+0000", 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10837039 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554582+0000", 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000254906 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.556153+0000", 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015706660000000001 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659815+0000", 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10366275 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.828747+0000", 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16893206199999999 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.828873+0000", 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012549100000000001 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.495 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:429 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.9606742069999998, 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51405278700000001, 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 429 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445677+0000", 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5260000000000001e-06 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445685+0000", 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445703+0000", 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8295999999999999e-05 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445718+0000", 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5383000000000002e-05 2024-04-18T21:52:18.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545773+0000", 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100054752 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545785+0000", 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1783000000000001e-05 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557166+0000", 2024-04-18T21:52:18.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011381578 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557443+0000", 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027702800000000002 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.558675+0000", 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012320479999999999 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:18.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659884+0000", 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101208162 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959511+0000", 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29962765699999999 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959729+0000", 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021733700000000001 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.501 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.4.19:7 3.as0 3:5bb4af71:::smithi09731216-7:head [assert-version v11,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.8516372940000001, 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40581794100000002, 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554716+0000", 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999999e-06 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554731+0000", 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4722999999999999e-05 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554766+0000", 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5216000000000003e-05 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554789+0000", 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2495999999999999e-05 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.749822+0000", 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19503310099999999 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.749982+0000", 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016067800000000001 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.960531+0000", 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.210548227 2024-04-18T21:52:18.505 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.506 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.506 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.506 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.506 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.506 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:18.506 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:18.507 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:18.507 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:18.507 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:18.508 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:18.508 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:18.508 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:18.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-18T21:52:18.511 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:18.507+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:18.584 INFO:tasks.rados.rados.0.smithi097.stdout:185: expect (ObjNum 66 snap 4 seq_num 66) 2024-04-18T21:52:18.611 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:18.607+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:18.641 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=200, last_complete=62'44, committed=1, applied=1))", 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.572041+0000", 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.068508637999999997, 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.068549657999999999, 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572041+0000", 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572041+0000", 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572041+0000", 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.39e-07 2024-04-18T21:52:18.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572045+0000", 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6449999999999998e-06 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572055+0000", 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7070000000000004e-06 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572061+0000", 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9070000000000001e-06 2024-04-18T21:52:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=201, last_complete=62'45, committed=1, applied=1))", 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.572071+0000", 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.068478082999999995, 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.068605892000000002, 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572071+0000", 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572071+0000", 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572072+0000", 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8000000000000002e-07 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572074+0000", 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.402e-06 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572083+0000", 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8340000000000005e-06 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572090+0000", 2024-04-18T21:52:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.64e-06 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=202, last_complete=62'47, committed=1, applied=1))", 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.572149+0000", 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.068400187000000001, 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.068605284000000002, 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.648 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572149+0000", 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572149+0000", 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572150+0000", 2024-04-18T21:52:18.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2500000000000001e-07 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572161+0000", 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0906e-05 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572169+0000", 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8850000000000005e-06 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572177+0000", 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7170000000000008e-06 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=199, last_complete=62'43, committed=1, applied=1))", 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.574220+0000", 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.066329582999999998, 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066612136000000002, 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.651 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574220+0000", 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574220+0000", 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574220+0000", 2024-04-18T21:52:18.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9800000000000004e-07 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574224+0000", 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6820000000000001e-06 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574233+0000", 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4339999999999998e-06 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574240+0000", 2024-04-18T21:52:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9269999999999999e-06 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=200, last_complete=62'44, committed=1, applied=1))", 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.574251+0000", 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.066298130999999996, 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066655281999999996, 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.654 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574251+0000", 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574251+0000", 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574252+0000", 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3299999999999998e-07 2024-04-18T21:52:18.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574254+0000", 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2740000000000002e-06 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574262+0000", 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7219999999999999e-06 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574268+0000", 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3210000000000001e-06 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=201, last_complete=62'45, committed=1, applied=1))", 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.574277+0000", 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.066272800000000007, 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066705287000000002, 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574277+0000", 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574277+0000", 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574277+0000", 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7300000000000002e-07 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574279+0000", 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5229999999999998e-06 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574287+0000", 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9629999999999998e-06 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574294+0000", 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4529999999999999e-06 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.659 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=202, last_complete=62'47, committed=1, applied=1))", 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.575848+0000", 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.064701147000000001, 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.065207669999999995, 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.575848+0000", 2024-04-18T21:52:18.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.575848+0000", 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.575849+0000", 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4300000000000001e-07 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.575854+0000", 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5959999999999997e-06 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.575867+0000", 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3641e-05 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.575878+0000", 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0383999999999999e-05 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.662 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=200, last_complete=62'44, committed=1, applied=1))", 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.568775+0000", 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.071774342000000005, 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.072356116999999998, 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568775+0000", 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568775+0000", 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568776+0000", 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4099999999999998e-07 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568785+0000", 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8270000000000004e-06 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568794+0000", 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.217e-06 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568802+0000", 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0239999999999997e-06 2024-04-18T21:52:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=201, last_complete=62'45, committed=1, applied=1))", 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.568813+0000", 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.071736219000000004, 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.072391940000000002, 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.666 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568813+0000", 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568813+0000", 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568814+0000", 2024-04-18T21:52:18.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3000000000000002e-07 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568816+0000", 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.542e-06 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568823+0000", 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3649999999999998e-06 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.568830+0000", 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1990000000000003e-06 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=202, last_complete=62'47, committed=1, applied=1))", 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.569877+0000", 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.070672619000000006, 2024-04-18T21:52:18.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071402959000000002, 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.569877+0000", 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.569877+0000", 2024-04-18T21:52:18.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.569878+0000", 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.02e-07 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.569883+0000", 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7010000000000001e-06 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.569900+0000", 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6725999999999999e-05 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.569910+0000", 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5279999999999992e-06 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.672 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=200, last_complete=62'44, committed=1, applied=1))", 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.570987+0000", 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.069561998999999999, 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.070388093999999998, 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.570987+0000", 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.570987+0000", 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.570988+0000", 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2500000000000004e-07 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.570995+0000", 2024-04-18T21:52:18.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5370000000000003e-06 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571011+0000", 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6600999999999999e-05 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571024+0000", 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2566e-05 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.675 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=201, last_complete=62'45, committed=1, applied=1))", 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.571044+0000", 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.069505474999999997, 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.070405832000000002, 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571044+0000", 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571044+0000", 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571044+0000", 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.22e-07 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571047+0000", 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1140000000000002e-06 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571058+0000", 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0944000000000001e-05 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571071+0000", 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2391e-05 2024-04-18T21:52:18.678 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=199, last_complete=62'43, committed=1, applied=1))", 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.571864+0000", 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.068685357000000002, 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069660203000000004, 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571864+0000", 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571864+0000", 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571865+0000", 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.316e-06 2024-04-18T21:52:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571870+0000", 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0470000000000001e-06 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571885+0000", 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4528e-05 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571896+0000", 2024-04-18T21:52:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0744000000000001e-05 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=200, last_complete=62'44, committed=1, applied=1))", 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.571915+0000", 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.068634708000000003, 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069683509000000005, 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.682 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571915+0000", 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571915+0000", 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571915+0000", 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5499999999999998e-07 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571919+0000", 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.534e-06 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571929+0000", 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9299999999999998e-06 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571936+0000", 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7849999999999992e-06 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=201, last_complete=62'45, committed=1, applied=1))", 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.571948+0000", 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.068601638000000006, 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069724595, 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.685 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571948+0000", 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571948+0000", 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571948+0000", 2024-04-18T21:52:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1600000000000002e-07 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571951+0000", 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9749999999999999e-06 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571960+0000", 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4819999999999994e-06 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571966+0000", 2024-04-18T21:52:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8940000000000004e-06 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 62/58 ECSubWriteReply(tid=199, last_complete=62'43, committed=1, applied=1))", 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.572009+0000", 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.068540114999999999, 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069737203999999997, 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572009+0000", 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572009+0000", 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572010+0000", 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7300000000000001e-07 2024-04-18T21:52:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572012+0000", 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6800000000000002e-06 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572021+0000", 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5259999999999993e-06 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572029+0000", 2024-04-18T21:52:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7700000000000001e-06 2024-04-18T21:52:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:52:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 16 2024-04-18T21:52:18.691 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:18.692 INFO:tasks.rados.rados.0.smithi097.stdout:187: finishing write tid 1 to smithi09731216-45 2024-04-18T21:52:18.692 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:18.737 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:18.735+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:18.769 INFO:tasks.rados.rados.0.smithi097.stdout:189: finishing rollback tid 0 to smithi09731216-48 2024-04-18T21:52:18.769 INFO:tasks.rados.rados.0.smithi097.stdout:189: finishing rollback tid 1 to smithi09731216-48 2024-04-18T21:52:18.779 INFO:tasks.rados.rados.0.smithi097.stdout:186: finishing rollback tid 0 to smithi09731216-36 2024-04-18T21:52:18.779 INFO:tasks.rados.rados.0.smithi097.stdout:186: finishing rollback tid 1 to smithi09731216-36 2024-04-18T21:52:18.780 INFO:tasks.rados.rados.0.smithi097.stdout:186: finishing rollback tid 2 to smithi09731216-36 2024-04-18T21:52:18.780 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-18T21:52:18.812 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:18.807+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:18.866 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:18.866 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:18.866 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:18.866 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:18.866 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:18.870 INFO:tasks.rados.rados.0.smithi097.stdout:189: finishing rollback tid 2 to smithi09731216-48 2024-04-18T21:52:18.870 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 48 v 45 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-18T21:52:18.876 INFO:tasks.rados.rados.0.smithi097.stdout:187: finishing write tid 2 to smithi09731216-45 2024-04-18T21:52:18.878 INFO:tasks.rados.rados.0.smithi097.stdout:187: finishing write tid 3 to smithi09731216-45 2024-04-18T21:52:18.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:18.925 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:18.923+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:18.971 INFO:tasks.rados.rados.0.smithi097.stdout:185: done (5 left) 2024-04-18T21:52:18.971 INFO:tasks.rados.rados.0.smithi097.stdout:186: done (4 left) 2024-04-18T21:52:18.971 INFO:tasks.rados.rados.0.smithi097.stdout:189: done (3 left) 2024-04-18T21:52:18.971 INFO:tasks.rados.rados.0.smithi097.stdout:190: done (2 left) 2024-04-18T21:52:18.971 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 28 current snap is 14 2024-04-18T21:52:18.971 INFO:tasks.rados.rados.0.smithi097.stdout:191: seq_num 86 ranges {1310720=688128,1998848=540672,2539520=229376} 2024-04-18T21:52:19.001 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:19.001 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.201323907999999, 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58281355499999998, 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.002 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 1 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798268+0000", 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5959999999999998e-06 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798283+0000", 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5495999999999999e-05 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798322+0000", 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8899999999999997e-05 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798347+0000", 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5510000000000001e-05 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886734+0000", 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088386079000000006 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886749+0000", 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5019e-05 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.078589+0000", 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19184037900000001 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079052+0000", 2024-04-18T21:52:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046354399999999998 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079551+0000", 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049870299999999997 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.083572+0000", 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040212609999999999 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.380910+0000", 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29733748900000001 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381009+0000", 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9025999999999997e-05 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381079+0000", 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9553000000000003e-05 2024-04-18T21:52:19.008 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.264514574000003, 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40660126499999999, 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 33 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735074+0000", 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735075+0000", 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4300000000000001e-07 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735081+0000", 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8669999999999999e-06 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735094+0000", 2024-04-18T21:52:19.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2813000000000001e-05 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735102+0000", 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4770000000000003e-06 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.737810+0000", 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027080089999999999 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.737953+0000", 2024-04-18T21:52:19.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014291900000000001 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.738162+0000", 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020895500000000001 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.940717+0000", 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20255461299999999 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141518+0000", 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20080082699999999 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141607+0000", 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9262e-05 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141676+0000", 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8579999999999997e-05 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.014 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.264473784000003, 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50698967299999997, 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.015 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 34 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735116+0000", 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735123+0000", 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3629999999999996e-06 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735156+0000", 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3030000000000001e-05 2024-04-18T21:52:19.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735167+0000", 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.13e-05 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939428+0000", 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20426096299999999 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939615+0000", 2024-04-18T21:52:19.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018654199999999999 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939841+0000", 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022621800000000001 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141747+0000", 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201905318 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.241958+0000", 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100211063 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242041+0000", 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3300000000000005e-05 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242105+0000", 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3895999999999999e-05 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.264398784999997, 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61088117099999995, 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 35 2024-04-18T21:52:19.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735191+0000", 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3699999999999999e-07 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735266+0000", 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4856000000000003e-05 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735279+0000", 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.36e-05 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735288+0000", 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0860000000000008e-06 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242298+0000", 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50700964999999998 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242568+0000", 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027023100000000001 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242974+0000", 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040533399999999998 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246576+0000", 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036026919999999998 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345792+0000", 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099215503999999996 2024-04-18T21:52:19.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345953+0000", 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161189 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346071+0000", 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118192 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.264286292000001, 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61108955600000003, 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 36 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.027 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:19.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.23e-07 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735310+0000", 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7560000000000002e-06 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735323+0000", 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735337+0000", 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4049999999999999e-05 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244243+0000", 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50890638099999996 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244422+0000", 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000178495 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244638+0000", 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216562 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.247950+0000", 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033121750000000001 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346232+0000", 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098281564000000002 2024-04-18T21:52:19.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346324+0000", 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2042999999999999e-05 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346392+0000", 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8052999999999994e-05 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.264237272999999, 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71224671699999997, 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 37 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.033 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:19.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0999999999999998e-07 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735358+0000", 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2639999999999999e-06 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735367+0000", 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735375+0000", 2024-04-18T21:52:19.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5360000000000004e-06 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245345+0000", 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50996983699999998 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245615+0000", 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027018099999999998 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246022+0000", 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000407394 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.248536+0000", 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025132330000000001 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447444+0000", 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198908419 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447532+0000", 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8010000000000006e-05 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447598+0000", 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6383000000000005e-05 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:52:19.038 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.35083676, 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61732750999999997, 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 117 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648753+0000", 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648758+0000", 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648768+0000", 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.414e-06 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648777+0000", 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.363e-06 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863479+0000", 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214701524 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863596+0000", 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117039 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863810+0000", 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021416800000000001 2024-04-18T21:52:19.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.864846+0000", 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001036319 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265892+0000", 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40104641899999999 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265995+0000", 2024-04-18T21:52:19.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010209600000000001 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.266080+0000", 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5049000000000005e-05 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:52:19.044 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.693925520000001, 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.509317829, 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 158 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305665+0000", 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5230000000000001e-06 2024-04-18T21:52:19.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305786+0000", 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012069 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305805+0000", 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9836000000000001e-05 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305821+0000", 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6030000000000001e-05 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307593+0000", 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017715299999999999 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307838+0000", 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024551800000000002 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.308174+0000", 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033593400000000003 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413731+0000", 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105556865 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814825+0000", 2024-04-18T21:52:19.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40109329500000002 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814911+0000", 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6138999999999996e-05 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814981+0000", 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0469000000000006e-05 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.693740464000001, 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50938204099999995, 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 159 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.051 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305849+0000", 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0559999999999999e-06 2024-04-18T21:52:19.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305856+0000", 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9600000000000003e-06 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305867+0000", 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0910000000000001e-05 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305879+0000", 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.147e-05 2024-04-18T21:52:19.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309071+0000", 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031920859999999998 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309303+0000", 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023236200000000001 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309671+0000", 2024-04-18T21:52:19.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036798000000000001 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416908+0000", 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107236578 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815085+0000", 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39817670700000002 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815165+0000", 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0196000000000003e-05 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815230+0000", 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5735999999999995e-05 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:19.056 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.693697161000003, 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50958771300000005, 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 160 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305898+0000", 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1359999999999996e-06 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305907+0000", 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5300000000000002e-06 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305915+0000", 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1869999999999997e-06 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310667+0000", 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047516800000000003 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310895+0000", 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000227875 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.311257+0000", 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000361737 2024-04-18T21:52:19.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416933+0000", 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10567686699999999 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815344+0000", 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39841079899999998 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815421+0000", 2024-04-18T21:52:19.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7229000000000002e-05 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815479+0000", 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7973000000000002e-05 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.693662498999998, 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60955713600000005, 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 161 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305927+0000", 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306056+0000", 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012895300000000001 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306068+0000", 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306077+0000", 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9430000000000006e-06 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412179+0000", 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106101709 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412267+0000", 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8251999999999995e-05 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412388+0000", 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121136 2024-04-18T21:52:19.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420472+0000", 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080833510000000008 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915141+0000", 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49466944899999998 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915356+0000", 2024-04-18T21:52:19.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215305 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915484+0000", 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012713500000000001 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.693498675999997, 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61031149600000001, 2024-04-18T21:52:19.068 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 162 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306091+0000", 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306097+0000", 2024-04-18T21:52:19.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6300000000000003e-06 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306106+0000", 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2199999999999998e-06 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306116+0000", 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8870000000000004e-06 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413869+0000", 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107753534 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414135+0000", 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026615200000000002 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414518+0000", 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038217299999999997 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420492+0000", 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059740879999999998 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915940+0000", 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49544827600000002 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916143+0000", 2024-04-18T21:52:19.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000202752 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916402+0000", 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025901100000000002 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.693462306000001, 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61064147300000005, 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 163 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.06e-07 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306136+0000", 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6999999999999997e-06 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306146+0000", 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8439999999999993e-06 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306154+0000", 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-04-18T21:52:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415593+0000", 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109439392 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415833+0000", 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000240101 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416223+0000", 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000389804 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420508+0000", 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042851699999999996 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916537+0000", 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49602917499999999 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916669+0000", 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179899999999999 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916768+0000", 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8879e-05 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.693423772999999, 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61096294100000004, 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 164 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.081 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306166+0000", 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306275+0000", 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109749 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306287+0000", 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1182999999999999e-05 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306298+0000", 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1487e-05 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417044+0000", 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110745818 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417282+0000", 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000237712 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417665+0000", 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000383194 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420523+0000", 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028578010000000001 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916880+0000", 2024-04-18T21:52:19.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49635731900000002 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917054+0000", 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017436200000000001 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917128+0000", 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3693000000000001e-05 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.693277778000002, 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70940268200000001, 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 165 2024-04-18T21:52:19.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306312+0000", 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306317+0000", 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0699999999999997e-06 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306327+0000", 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7030000000000001e-06 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306349+0000", 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2160000000000001e-05 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419233+0000", 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112884159 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419499+0000", 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265865 2024-04-18T21:52:19.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419851+0000", 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035253999999999998 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.421040+0000", 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011886380000000001 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:19.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015628+0000", 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59458825299999996 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015677+0000", 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8470000000000002e-05 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015714+0000", 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7123999999999999e-05 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:437 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~638976] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.475053+0000", 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.5245355810000003, 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.407913252, 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 437 2024-04-18T21:52:19.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.475053+0000", 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.475053+0000", 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.475056+0000", 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8100000000000002e-06 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.476084+0000", 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001028321 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.476118+0000", 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3130000000000003e-05 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.476134+0000", 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6446000000000001e-05 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.521238+0000", 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.045103754000000003 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.521540+0000", 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000301902 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.524173+0000", 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026329220000000002 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.782083+0000", 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.25791004899999997 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.882818+0000", 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100735441 2024-04-18T21:52:19.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.882967+0000", 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014847699999999999 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:438 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~622592] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.5175142739999998, 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60190392599999998, 2024-04-18T21:52:19.098 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 438 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482077+0000", 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3429999999999998e-06 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482901+0000", 2024-04-18T21:52:19.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082364099999999996 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482933+0000", 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2128999999999997e-05 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482949+0000", 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5872999999999999e-05 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676460+0000", 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19351131799999999 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676647+0000", 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018723700000000001 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.678595+0000", 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00194812 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.782086+0000", 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10349056299999999 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083745+0000", 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30165924199999999 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083979+0000", 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023346 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.103 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:439 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~147456] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.5158967890000001, 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70156516499999999, 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:19.104 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 439 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483694+0000", 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2129999999999998e-06 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483902+0000", 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020762600000000001 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483934+0000", 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2140000000000001e-05 2024-04-18T21:52:19.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483950+0000", 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5792999999999999e-05 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779688+0000", 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29573833799999999 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779887+0000", 2024-04-18T21:52:19.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019900000000000001 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.781116+0000", 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001229071 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.882567+0000", 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101451022 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185116+0000", 2024-04-18T21:52:19.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.302548704 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185257+0000", 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014125800000000001 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:440 3.fs0 3:fcd1a062:::smithi09731216-16:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.5156129140000001, 2024-04-18T21:52:19.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.10743576, 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 440 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:19.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483977+0000", 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7000000000000003e-07 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483982+0000", 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4e-06 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483994+0000", 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.134e-05 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484003+0000", 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1169999999999993e-06 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186403+0000", 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70240069199999999 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186624+0000", 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000220523 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.187505+0000", 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088087199999999999 2024-04-18T21:52:19.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188877+0000", 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001371723 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591138+0000", 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40226145699999999 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591412+0000", 2024-04-18T21:52:19.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027366599999999999 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:441 3.fs0 3:fcd1a062:::smithi09731216-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.5155736710000003, 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.413060682, 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:19.115 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 441 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:19.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484016+0000", 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8299999999999996e-07 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484019+0000", 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3069999999999998e-06 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484038+0000", 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978999999999999e-05 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484047+0000", 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5939999999999994e-06 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188125+0000", 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70407843599999997 2024-04-18T21:52:19.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188355+0000", 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022941899999999999 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692687+0000", 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50433201000000005 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692919+0000", 2024-04-18T21:52:19.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023258299999999999 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.897076+0000", 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20415657100000001 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:19.120 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:19.122 INFO:tasks.rados.rados.0.smithi097.stdout:191: writing smithi09731216-28 from 1310720 to 1998848 tid 1 2024-04-18T21:52:19.122 INFO:tasks.rados.rados.0.smithi097.stdout:191: writing smithi09731216-28 from 1998848 to 2539520 tid 2 2024-04-18T21:52:19.122 INFO:tasks.rados.rados.0.smithi097.stdout:191: writing smithi09731216-28 from 2539520 to 2768896 tid 3 2024-04-18T21:52:19.122 INFO:tasks.rados.rados.0.smithi097.stdout:192: snap_create 2024-04-18T21:52:19.122 INFO:tasks.rados.rados.0.smithi097.stdout:187: finishing write tid 4 to smithi09731216-45 2024-04-18T21:52:19.123 INFO:tasks.rados.rados.0.smithi097.stdout:188: expect (ObjNum 74 snap 8 seq_num 74) 2024-04-18T21:52:19.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:19.123 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:19.121+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:19.223 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:19.221+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:19.253 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:19.253 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:19.253 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:19.254 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:19.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:19.323 INFO:tasks.rados.rados.0.smithi097.stdout:191: finishing write tid 1 to smithi09731216-28 2024-04-18T21:52:19.323 INFO:tasks.rados.rados.0.smithi097.stdout:191: finishing write tid 2 to smithi09731216-28 2024-04-18T21:52:19.324 INFO:tasks.rados.rados.0.smithi097.stdout:191: finishing write tid 3 to smithi09731216-28 2024-04-18T21:52:19.356 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:19.357+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:19.381 INFO:tasks.rados.rados.0.smithi097.stdout:187: finishing write tid 5 to smithi09731216-45 2024-04-18T21:52:19.381 INFO:tasks.rados.rados.0.smithi097.stdout:187: oid 45 updating version 0 to 50 2024-04-18T21:52:19.382 INFO:tasks.rados.rados.0.smithi097.stdout:187: oid 45 version 50 is already newer than 49 2024-04-18T21:52:19.382 INFO:tasks.rados.rados.0.smithi097.stdout:187: oid 45 updating version 50 to 51 2024-04-18T21:52:19.382 INFO:tasks.rados.rados.0.smithi097.stdout:187: oid 45 version 51 is already newer than 48 2024-04-18T21:52:19.382 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 45 v 51 (ObjNum 84 snap 14 seq_num 84) dirty exists 2024-04-18T21:52:19.382 INFO:tasks.rados.rados.0.smithi097.stdout:187: left oid 45 (ObjNum 84 snap 14 seq_num 84) 2024-04-18T21:52:19.424 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:19.425+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:19.425 INFO:tasks.rados.rados.0.smithi097.stdout:191: finishing write tid 4 to smithi09731216-28 2024-04-18T21:52:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:19.431 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:19.431 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:19.431 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:19.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:19.458 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is set 2024-04-18T21:52:19.462 INFO:tasks.rados.rados.0.smithi097.stdout:187: done (3 left) 2024-04-18T21:52:19.462 INFO:tasks.rados.rados.0.smithi097.stdout:188: done (2 left) 2024-04-18T21:52:19.463 INFO:tasks.rados.rados.0.smithi097.stdout:192: done (1 left) 2024-04-18T21:52:19.463 INFO:tasks.rados.rados.0.smithi097.stdout:193: snap_remove snap 7 2024-04-18T21:52:19.532 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:19.533+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:19.615 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:19.615 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:19.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:19.615 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:19.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.047409469000002, 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201960743, 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566449+0000", 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.77e-06 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566495+0000", 2024-04-18T21:52:19.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5488999999999998e-05 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566536+0000", 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0756000000000001e-05 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.666638+0000", 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102466 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766859+0000", 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10022078500000001 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766875+0000", 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5806000000000001e-05 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768312+0000", 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014373389999999999 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768408+0000", 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6331999999999999e-05 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.4s4 30/26 ECSubWrite(tid=13, reqid=client.4509.0:98, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.936882732000001, 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19094267400000001, 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676974+0000", 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.676976+0000", 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9e-06 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677025+0000", 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8730000000000003e-05 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677075+0000", 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0405999999999998e-05 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.677101+0000", 2024-04-18T21:52:19.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5777000000000001e-05 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.774116+0000", 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.097015016999999995 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.774145+0000", 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9009999999999998e-05 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867877+0000", 2024-04-18T21:52:19.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093731420999999995 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.867917+0000", 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0413000000000001e-05 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:118 3.17s0 3:eda737c1:::smithi09731216-40:head [append 0~671744,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.009166239999999, 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1801127490000001, 2024-04-18T21:52:19.624 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 118 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604693+0000", 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2970000000000002e-06 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.605995+0000", 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001302276 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606021+0000", 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5320000000000002e-05 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606036+0000", 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5133000000000001e-05 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606232+0000", 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000196672 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606237+0000", 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7199999999999997e-06 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484281+0000", 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.87804420999999999 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484592+0000", 2024-04-18T21:52:19.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031024299999999998 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.487837+0000", 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003245571 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490891+0000", 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030539719999999998 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784613+0000", 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.293722237 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784804+0000", 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019009800000000001 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:119 3.17s0 3:eda737c1:::smithi09731216-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.007784391000001, 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.179191986, 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 119 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606074+0000", 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.257e-06 2024-04-18T21:52:19.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606080+0000", 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0100000000000001e-06 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606095+0000", 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5413e-05 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606104+0000", 2024-04-18T21:52:19.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0899999999999994e-06 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606286+0000", 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000181552 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606287+0000", 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.026e-06 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493720+0000", 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88743291700000004 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493872+0000", 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015190200000000001 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495103+0000", 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00123149 2024-04-18T21:52:19.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.496855+0000", 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017518189999999999 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785096+0000", 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.28824129900000001 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785265+0000", 2024-04-18T21:52:19.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016821100000000001 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:120 3.17s0 3:eda737c1:::smithi09731216-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.007715261000001, 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4107226310000001, 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 120 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606143+0000", 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.16e-07 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606148+0000", 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4569999999999996e-06 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606163+0000", 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503e-05 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606175+0000", 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2036e-05 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706293+0000", 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100117875 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706297+0000", 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.126e-06 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495621+0000", 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78932349599999996 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495757+0000", 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013642600000000001 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785995+0000", 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29023745499999998 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.786127+0000", 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013212200000000001 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.016864+0000", 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.23073769199999999 2024-04-18T21:52:19.642 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=0, last_complete=32'1, committed=1, applied=1))", 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 31.030039685999999, 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.112466288, 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583817+0000", 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583819+0000", 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.604e-06 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583830+0000", 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0883000000000001e-05 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.583857+0000", 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7053e-05 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683949+0000", 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100091869 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696264+0000", 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012315196 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696284+0000", 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9683000000000001e-05 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.7s0,GRANT,e:32,from:3(1),reservation_nonce:3)", 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:52:19.646 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.964605733999999, 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14739324400000001, 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649251+0000", 2024-04-18T21:52:19.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649253+0000", 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0760000000000001e-06 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649265+0000", 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1523e-05 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649299+0000", 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3662999999999999e-05 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.649316+0000", 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7197000000000002e-05 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796388+0000", 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14707189000000001 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796474+0000", 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6038999999999994e-05 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796645+0000", 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017085599999999999 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.650 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.17s0 32/31 ECSubRead(tid=3, to_read={3:eda737c1:::smithi09731216-40:head=0,4096,0}, subchunks={3:eda737c1:::smithi09731216-40:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.826674260000001, 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.210779366, 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787250+0000", 2024-04-18T21:52:19.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7152999999999999e-05 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887353+0000", 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102647 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997704+0000", 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110351217 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997962+0000", 2024-04-18T21:52:19.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000258349 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.17s0 32/31 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.817379255999999, 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200936119, 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796479+0000", 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8999999999999995e-07 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796491+0000", 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2065000000000001e-05 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796502+0000", 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.116e-05 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796509+0000", 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7170000000000008e-06 2024-04-18T21:52:19.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997338+0000", 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20082881799999999 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997414+0000", 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5568999999999994e-05 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:19.657 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.817212536, 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200846881, 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796645+0000", 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8999999999999998e-07 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796647+0000", 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5239999999999999e-06 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796655+0000", 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796660+0000", 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4879999999999998e-06 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997463+0000", 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20080266799999999 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997491+0000", 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8192e-05 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.816691984999999, 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20052022, 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.661 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797166+0000", 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5300000000000004e-07 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797170+0000", 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.233e-06 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797183+0000", 2024-04-18T21:52:19.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3454e-05 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797191+0000", 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5769999999999996e-06 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997659+0000", 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20046852900000001 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997685+0000", 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5774000000000001e-05 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.7s0,GRANT,e:32,from:4(6),reservation_nonce:4)", 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.816320171000001, 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20023016099999999, 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797537+0000", 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797544+0000", 2024-04-18T21:52:19.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1670000000000002e-06 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797551+0000", 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9959999999999996e-06 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797574+0000", 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.321e-05 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797587+0000", 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.235e-05 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997665+0000", 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200078382 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997695+0000", 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0499999999999999e-05 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997767+0000", 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1556000000000004e-05 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.726865370999999, 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110630505, 2024-04-18T21:52:19.669 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886992+0000", 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.886993+0000", 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.539e-06 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887021+0000", 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7529e-05 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887046+0000", 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4873000000000001e-05 2024-04-18T21:52:19.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887060+0000", 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4324e-05 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997594+0000", 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11053360800000001 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997622+0000", 2024-04-18T21:52:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8632e-05 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/31 ECSubWrite(tid=28, reqid=client.4509.0:304, at_version=41'12, trim_to=0'0, roll_forward_to=32'8))", 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.444489693000001, 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201742379, 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169369+0000", 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6339999999999999e-06 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169413+0000", 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3860999999999999e-05 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169487+0000", 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4578000000000003e-05 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.269583+0000", 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100095613 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369850+0000", 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10026700299999999 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369869+0000", 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9114e-05 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371022+0000", 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001152943 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371110+0000", 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7633000000000006e-05 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/26 ECSubWrite(tid=56, reqid=osd.10.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'25))", 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 20.828413669, 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201706881, 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.677 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785445+0000", 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7999999999999999e-06 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785499+0000", 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3730999999999998e-05 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785543+0000", 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3961000000000001e-05 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785568+0000", 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.561e-05 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.985970+0000", 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200401511 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.986000+0000", 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0224000000000001e-05 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987085+0000", 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001085145 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987150+0000", 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4899000000000003e-05 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.681 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.17s0,GRANT,e:42,from:10(3),reservation_nonce:10)", 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 20.333268639, 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200838461, 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280590+0000", 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.587e-06 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280598+0000", 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5450000000000003e-06 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280627+0000", 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8141e-05 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.380804+0000", 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10017703 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481049+0000", 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10024543 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481088+0000", 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8720999999999997e-05 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481427+0000", 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000339007 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 53/51 ECSubWrite(tid=2, reqid=client.4509.0:390, at_version=0'0, trim_to=0'0, roll_forward_to=53'7))", 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 9.9152338069999999, 2024-04-18T21:52:19.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201345094, 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698625+0000", 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2869999999999999e-06 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698665+0000", 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9743000000000002e-05 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698719+0000", 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4228999999999999e-05 2024-04-18T21:52:19.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.798874+0000", 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100154506 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.799005+0000", 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013080200000000001 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.799030+0000", 2024-04-18T21:52:19.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5058e-05 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.899930+0000", 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100900189 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.899968+0000", 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8279999999999999e-05 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.2s5 58/31 ECSubWrite(tid=87, reqid=client.4509.0:422, at_version=58'6, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.9556711279999996, 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201964437, 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:19.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658187+0000", 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5069999999999999e-06 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658288+0000", 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010070500000000001 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658315+0000", 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7348e-05 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658331+0000", 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5559000000000001e-05 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.758532+0000", 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100200734 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.758549+0000", 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6823000000000001e-05 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860062+0000", 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101513567 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860150+0000", 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8194000000000003e-05 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.2s5 58/31 ECSubWrite(tid=88, reqid=client.4509.0:423, at_version=58'7, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.9554868699999997, 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20184664099999999, 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.694 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658371+0000", 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5400000000000001e-07 2024-04-18T21:52:19.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658390+0000", 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8907e-05 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658404+0000", 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3974999999999999e-05 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658414+0000", 2024-04-18T21:52:19.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0156000000000001e-05 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.759248+0000", 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10083389299999999 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.759263+0000", 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5101e-05 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860159+0000", 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10089624599999999 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860217+0000", 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7809000000000002e-05 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.fs7 62/26 ECSubWrite(tid=203, reqid=osd.7.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=62'26))", 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:19.698 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.037650411, 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20165132299999999, 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:19.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576209+0000", 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9300000000000002e-06 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576234+0000", 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5287999999999999e-05 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:19.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576260+0000", 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5808000000000001e-05 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.676352+0000", 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10009247 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.776560+0000", 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100207532 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:19.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.776579+0000", 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8936e-05 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.777791+0000", 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00121206 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.777858+0000", 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7298999999999994e-05 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.702 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:19.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:19.703 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:19.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:19.705 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:19.699+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:19.805 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:19.803+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:19.874 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:19.874 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:19.874 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:19.874 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:19.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:19.905 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:19.903+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:20.006 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:20.005+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:20.011 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:20.011 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:20.011 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:20.011 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:20.011 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:20.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:20.107 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:20.107+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:20.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=7, reqid=client.4509.0:94, at_version=30'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.53005546, 2024-04-18T21:52:20.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201169337, 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:20.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661077+0000", 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4600000000000002e-06 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661394+0000", 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031720499999999998 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661424+0000", 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9822999999999999e-05 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661441+0000", 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7496e-05 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661661+0000", 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000219602 2024-04-18T21:52:20.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661676+0000", 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5333e-05 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862187+0000", 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200510299 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.196 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862244+0000", 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7119e-05 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.529652153000001, 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201435581, 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.197 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661479+0000", 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0839999999999999e-06 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661509+0000", 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0389999999999999e-05 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661529+0000", 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0219000000000001e-05 2024-04-18T21:52:20.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661553+0000", 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3836999999999999e-05 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861851+0000", 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20029786299999999 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861866+0000", 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4769999999999999e-05 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862836+0000", 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097006600000000001 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862913+0000", 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7352000000000001e-05 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.201 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.8s4 30/26 ECSubWrite(tid=12, reqid=client.4509.0:104, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.321571851999998, 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11605781900000001, 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.202 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869560+0000", 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-06 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869593+0000", 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3216000000000002e-05 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869624+0000", 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1164000000000002e-05 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969744+0000", 2024-04-18T21:52:20.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100119518 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969958+0000", 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021381800000000001 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969985+0000", 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7376999999999999e-05 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985510+0000", 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015525357 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985616+0000", 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105479 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.206 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.17s4 32/31 ECSubWrite(tid=1, reqid=client.4509.0:119, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 31.696001292999998, 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18978218799999999, 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495130+0000", 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0860000000000001e-06 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495154+0000", 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4457999999999999e-05 2024-04-18T21:52:20.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495174+0000", 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9426999999999999e-05 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495188+0000", 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4813e-05 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683568+0000", 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.188379871 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683583+0000", 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5227e-05 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.684845+0000", 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012619560000000001 2024-04-18T21:52:20.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.684911+0000", 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5350000000000003e-05 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.14s6 32/31 ECSubWrite(tid=20, reqid=client.4509.0:125, at_version=32'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 31.594625143000002, 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20154013100000001, 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.211 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3799999999999997e-07 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596521+0000", 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5455999999999999e-05 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596532+0000", 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0881e-05 2024-04-18T21:52:20.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696606+0000", 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100073932 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796760+0000", 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10015476600000001 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796771+0000", 2024-04-18T21:52:20.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0278999999999999e-05 2024-04-18T21:52:20.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797921+0000", 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001150657 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.798045+0000", 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012362199999999999 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 41/31 ECSubWrite(tid=26, reqid=client.4509.0:302, at_version=41'10, trim_to=0'0, roll_forward_to=32'8))", 2024-04-18T21:52:20.218 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.069045+0000", 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 22.122084849, 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106695074, 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069045+0000", 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069045+0000", 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069046+0000", 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0699999999999996e-07 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069106+0000", 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0316000000000003e-05 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069142+0000", 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5592000000000001e-05 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.069190+0000", 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8321e-05 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169610+0000", 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100419604 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169637+0000", 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7277e-05 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.175645+0000", 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0060080450000000001 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.175740+0000", 2024-04-18T21:52:20.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5211999999999996e-05 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es6 41/26 ECSubWrite(tid=30, reqid=client.4509.0:316, at_version=41'18, trim_to=0'0, roll_forward_to=38'15))", 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.995229288000001, 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200900518, 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195902+0000", 2024-04-18T21:52:20.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3699999999999999e-07 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195937+0000", 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5682000000000002e-05 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195960+0000", 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2940999999999999e-05 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296055+0000", 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100094687 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296167+0000", 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011265900000000001 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296183+0000", 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5605000000000001e-05 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.396721+0000", 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10053751700000001 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.396801+0000", 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0489999999999997e-05 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.227 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.as2 41/26 ECSubWrite(tid=14, reqid=client.4509.0:309, at_version=41'10, trim_to=0'0, roll_forward_to=38'8))", 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.280352+0000", 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.910777468999999, 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103365524, 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280352+0000", 2024-04-18T21:52:20.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280352+0000", 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280354+0000", 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.753e-06 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280592+0000", 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023777800000000001 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280621+0000", 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9166000000000002e-05 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.380731+0000", 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10010949600000001 2024-04-18T21:52:20.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.380866+0000", 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013568 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.380892+0000", 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5573999999999999e-05 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.383650+0000", 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027578720000000002 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.383718+0000", 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8205000000000002e-05 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.19s5 41/31 ECSubWrite(tid=38, reqid=client.4509.0:318, at_version=41'9, trim_to=0'0, roll_forward_to=33'7))", 2024-04-18T21:52:20.232 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.815840050999999, 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.199621414, 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.233 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375292+0000", 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7069999999999999e-06 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375331+0000", 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9774999999999998e-05 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375361+0000", 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.974e-05 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375379+0000", 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8017e-05 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375529+0000", 2024-04-18T21:52:20.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015042500000000001 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375547+0000", 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7414999999999998e-05 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.574854+0000", 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19930687499999999 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.574911+0000", 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7460000000000002e-05 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.17s4,RELEASE,e:42,from:8(0),reservation_nonce:0)", 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 20.403865819, 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20038272400000001, 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.237 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787265+0000", 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.02e-06 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787274+0000", 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.439e-06 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787295+0000", 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1121e-05 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787312+0000", 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7838000000000001e-05 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987569+0000", 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200256506 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987603+0000", 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4072000000000002e-05 2024-04-18T21:52:20.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987647+0000", 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3727999999999999e-05 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.11s3 47/45 ECSubRead(tid=45, to_read={3:8fad7925:::smithi09731216-43:head=0,262144,0}, subchunks={3:8fad7925:::smithi09731216-43:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 16.793393191, 2024-04-18T21:52:20.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20153923400000001, 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:20.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397739+0000", 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.367e-06 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397761+0000", 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1525000000000001e-05 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397804+0000", 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3488000000000001e-05 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.497972+0000", 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100167611 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.598288+0000", 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100316505 2024-04-18T21:52:20.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.599276+0000", 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00098773799999999994 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.985458+0000", 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 11.205672218, 2024-04-18T21:52:20.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103534346, 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985458+0000", 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985458+0000", 2024-04-18T21:52:20.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985459+0000", 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.73e-06 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985910+0000", 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00045051200000000003 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985956+0000", 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6519e-05 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.985970+0000", 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3998999999999999e-05 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.088869+0000", 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102898447 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.088992+0000", 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000123139 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.986048+0000", 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 11.205081565, 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103397726, 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.249 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986048+0000", 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986048+0000", 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986049+0000", 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9699999999999994e-07 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986623+0000", 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000573541 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986714+0000", 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1388e-05 2024-04-18T21:52:20.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986726+0000", 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1609e-05 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.089309+0000", 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10258281299999999 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.089446+0000", 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013737800000000001 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.986487+0000", 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 11.204643168, 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102748308, 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.253 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986487+0000", 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986487+0000", 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986488+0000", 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.79e-06 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.987372+0000", 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00088342200000000003 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.987400+0000", 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7827999999999999e-05 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087487+0000", 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100087129 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.089105+0000", 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001617977 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.089235+0000", 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013016199999999999 2024-04-18T21:52:20.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 11.204412826, 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20394542500000001, 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986718+0000", 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:52:20.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087949+0000", 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10123096600000001 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087978+0000", 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9116999999999999e-05 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188141+0000", 2024-04-18T21:52:20.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10016269799999999 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.190504+0000", 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023637250000000001 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.190662+0000", 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000158076 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.188600+0000", 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 11.002529945999999, 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103165013, 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188600+0000", 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188600+0000", 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188601+0000", 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.353e-06 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188823+0000", 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00022220700000000001 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188841+0000", 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7308000000000001e-05 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188855+0000", 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4079e-05 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.288469+0000", 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099613660000000007 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291765+0000", 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003296406 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(3.12s0 52/51 [PushOp(3:4c1ede14:::smithi09731216-49:head, version: 33'5, data_included: [0~204800], data_size: 204800, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:4c1ede14:::smithi09731216-49:head@33'5, size: 819200, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 819200, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-18T21:52:20.264 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.899727678, 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10897306900000001, 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:20.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291402+0000", 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.291489+0000", 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.721e-05 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.391642+0000", 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100152688 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.397779+0000", 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0061370160000000003 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.400375+0000", 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025961550000000002 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 53/51 ECSubWriteReply(tid=1, last_complete=53'7, committed=1, applied=1))", 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.672915239, 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.180820172, 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.268 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518217+0000", 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9199999999999998e-06 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518227+0000", 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0356e-05 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518255+0000", 2024-04-18T21:52:20.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8144e-05 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518274+0000", 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9218000000000001e-05 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.697313+0000", 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17903920600000001 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.699035+0000", 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017213280000000001 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 62/58 ECSubWrite(tid=200, reqid=client.4509.0:471, at_version=62'44, trim_to=0'0, roll_forward_to=61'41))", 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:18.466994+0000", 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.7241359969999999, 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10509159699999999, 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.272 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.466994+0000", 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.466994+0000", 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.466996+0000", 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7400000000000001e-06 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.467049+0000", 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3356999999999998e-05 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.467078+0000", 2024-04-18T21:52:20.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9275e-05 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.467103+0000", 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4624e-05 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.467499+0000", 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00039661999999999998 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.467555+0000", 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5429000000000001e-05 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.571945+0000", 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104389705 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.572085+0000", 2024-04-18T21:52:20.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014084699999999999 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 62/58 ECSubWrite(tid=201, reqid=client.4509.0:472, at_version=62'45, trim_to=0'0, roll_forward_to=61'41))", 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:18.469756+0000", 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.7213742409999999, 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102428334, 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.277 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.469756+0000", 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.469756+0000", 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.469757+0000", 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1799999999999999e-06 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.469795+0000", 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8297000000000003e-05 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.469820+0000", 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4811999999999999e-05 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.469835+0000", 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4982e-05 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.570050+0000", 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100215452 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.570082+0000", 2024-04-18T21:52:20.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1275000000000001e-05 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.572093+0000", 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020116689999999998 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.572184+0000", 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0667000000000005e-05 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.282 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.282 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.282 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:20.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:20.284 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:20.279+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:20.384 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:20.383+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:20.405 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:20.405 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:20.405 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:20.405 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:20.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:20.435 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:20.436 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:20.454 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-18T21:52:20.259664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54567456,"num_objects":75,"num_object_clones":23,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":4,"num_read":259,"num_read_kb":27215,"num_write":351,"num_write_kb":36162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":644,"ondisk_log_size":644,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":234,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":464844,"kb_used_data":111444,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1131997236,"statfs":{"total":1159641169920,"available":1159165169664,"internally_reserved":0,"allocated":114118656,"data_stored":124191576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,0,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2777088,"num_objects":2,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":6,"num_read_kb":5763,"num_write":9,"num_write_kb":1912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001336"},"pg_stats":[{"pgid":"3.19","version":"56'22","reported_seq":208,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827516+0000","last_change":"2024-04-18T21:52:18.468380+0000","last_active":"2024-04-18T21:52:19.827516+0000","last_peered":"2024-04-18T21:52:19.827516+0000","last_clean":"2024-04-18T21:52:19.827516+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:19.827516+0000","last_undegraded":"2024-04-18T21:52:19.827516+0000","last_fullsized":"2024-04-18T21:52:19.827516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:21.638148+0000","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0.101186106,"stat_sum":{"num_bytes":1376256,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.18","version":"51'20","reported_seq":201,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827463+0000","last_change":"2024-04-18T21:52:18.372220+0000","last_active":"2024-04-18T21:52:19.827463+0000","last_peered":"2024-04-18T21:52:19.827463+0000","last_clean":"2024-04-18T21:52:19.827463+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:19.827463+0000","last_undegraded":"2024-04-18T21:52:19.827463+0000","last_fullsized":"2024-04-18T21:52:19.827463+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.945044+0000","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0.0053765150000000001,"stat_sum":{"num_bytes":2523136,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2258,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.17","version":"51'8","reported_seq":182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.376323+0000","last_change":"2024-04-18T21:52:18.477294+0000","last_active":"2024-04-18T21:52:18.376323+0000","last_peered":"2024-04-18T21:52:18.376323+0000","last_clean":"2024-04-18T21:52:18.376323+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:18.376323+0000","last_undegraded":"2024-04-18T21:52:18.376323+0000","last_fullsized":"2024-04-18T21:52:18.376323+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:13.107163+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.10287663,"stat_sum":{"num_bytes":671744,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.16","version":"32'3","reported_seq":181,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.366807+0000","last_change":"2024-04-18T21:52:18.368049+0000","last_active":"2024-04-18T21:52:18.366807+0000","last_peered":"2024-04-18T21:52:18.366807+0000","last_clean":"2024-04-18T21:52:18.366807+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:18.366807+0000","last_undegraded":"2024-04-18T21:52:18.366807+0000","last_fullsized":"2024-04-18T21:52:18.366807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:06.462212+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.0015130409999999999,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.15","version":"51'9","reported_seq":187,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727434+0000","last_change":"2024-04-18T21:52:18.568579+0000","last_active":"2024-04-18T21:52:19.727434+0000","last_peered":"2024-04-18T21:52:19.727434+0000","last_clean":"2024-04-18T21:52:19.727434+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:19.727434+0000","last_undegraded":"2024-04-18T21:52:19.727434+0000","last_fullsized":"2024-04-18T21:52:19.727434+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.274776+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.00073016800000000001,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.14","version":"51'8","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.455291+0000","last_change":"2024-04-18T21:52:15.456411+0000","last_active":"2024-04-18T21:52:15.455291+0000","last_peered":"2024-04-18T21:52:15.455291+0000","last_clean":"2024-04-18T21:52:15.455291+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:15.455291+0000","last_undegraded":"2024-04-18T21:52:15.455291+0000","last_fullsized":"2024-04-18T21:52:15.455291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.605862+0000","scrub_duration":1515,"objects_trimmed":0,"snaptrim_duration":0.0017272139999999999,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.459239+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:15.459239+0000","last_peered":"2024-04-18T21:52:15.459239+0000","last_clean":"2024-04-18T21:52:15.459239+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:15.459239+0000","last_undegraded":"2024-04-18T21:52:15.459239+0000","last_fullsized":"2024-04-18T21:52:15.459239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'6","reported_seq":180,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.781520+0000","last_change":"2024-04-18T21:52:18.780498+0000","last_active":"2024-04-18T21:52:18.781520+0000","last_peered":"2024-04-18T21:52:18.781520+0000","last_clean":"2024-04-18T21:52:18.781520+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:18.781520+0000","last_undegraded":"2024-04-18T21:52:18.781520+0000","last_fullsized":"2024-04-18T21:52:18.781520+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.144353+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.413638432,"stat_sum":{"num_bytes":589824,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":91,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.729008+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:19.729008+0000","last_peered":"2024-04-18T21:52:19.729008+0000","last_clean":"2024-04-18T21:52:19.729008+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:19.729008+0000","last_undegraded":"2024-04-18T21:52:19.729008+0000","last_fullsized":"2024-04-18T21:52:19.729008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":171,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.476485+0000","last_change":"2024-04-18T21:52:18.572183+0000","last_active":"2024-04-18T21:52:18.476485+0000","last_peered":"2024-04-18T21:52:18.476485+0000","last_clean":"2024-04-18T21:52:18.476485+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:18.476485+0000","last_undegraded":"2024-04-18T21:52:18.476485+0000","last_fullsized":"2024-04-18T21:52:18.476485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:51.625973+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.096017020999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.928705+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:19.928705+0000","last_peered":"2024-04-18T21:52:19.928705+0000","last_clean":"2024-04-18T21:52:19.928705+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:19.928705+0000","last_undegraded":"2024-04-18T21:52:19.928705+0000","last_fullsized":"2024-04-18T21:52:19.928705+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.456707+0000","last_change":"2024-04-18T21:52:15.458162+0000","last_active":"2024-04-18T21:52:15.456707+0000","last_peered":"2024-04-18T21:52:15.456707+0000","last_clean":"2024-04-18T21:52:15.456707+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:15.456707+0000","last_undegraded":"2024-04-18T21:52:15.456707+0000","last_fullsized":"2024-04-18T21:52:15.456707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.146084+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.0033094909999999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3170,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.366253+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:18.366253+0000","last_peered":"2024-04-18T21:52:18.366253+0000","last_clean":"2024-04-18T21:52:18.366253+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:18.366253+0000","last_undegraded":"2024-04-18T21:52:18.366253+0000","last_fullsized":"2024-04-18T21:52:18.366253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"61'15","reported_seq":192,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827756+0000","last_change":"2024-04-18T21:52:18.369180+0000","last_active":"2024-04-18T21:52:19.827756+0000","last_peered":"2024-04-18T21:52:19.827756+0000","last_clean":"2024-04-18T21:52:19.827756+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:19.827756+0000","last_undegraded":"2024-04-18T21:52:19.827756+0000","last_fullsized":"2024-04-18T21:52:19.827756+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.420167+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.001025174,"stat_sum":{"num_bytes":868352,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.3","version":"59'84","reported_seq":257,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.763076+0000","last_change":"2024-04-18T21:52:15.765341+0000","last_active":"2024-04-18T21:52:15.763076+0000","last_peered":"2024-04-18T21:52:15.763076+0000","last_clean":"2024-04-18T21:52:15.763076+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:15.763076+0000","last_undegraded":"2024-04-18T21:52:15.763076+0000","last_fullsized":"2024-04-18T21:52:15.763076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:24.270053+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.28746011199999999,"stat_sum":{"num_bytes":10403840,"num_objects":11,"num_object_clones":4,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727403+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:19.727403+0000","last_peered":"2024-04-18T21:52:19.727403+0000","last_clean":"2024-04-18T21:52:19.727403+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:19.727403+0000","last_undegraded":"2024-04-18T21:52:19.727403+0000","last_fullsized":"2024-04-18T21:52:19.727403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"59'22","reported_seq":189,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.957919+0000","last_change":"2024-04-18T21:52:15.959178+0000","last_active":"2024-04-18T21:52:15.957919+0000","last_peered":"2024-04-18T21:52:15.957919+0000","last_clean":"2024-04-18T21:52:15.957919+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:15.957919+0000","last_undegraded":"2024-04-18T21:52:15.957919+0000","last_fullsized":"2024-04-18T21:52:15.957919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.113096+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.50073931699999996,"stat_sum":{"num_bytes":3866624,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727911+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:19.727911+0000","last_peered":"2024-04-18T21:52:19.727911+0000","last_clean":"2024-04-18T21:52:19.727911+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:19.727911+0000","last_undegraded":"2024-04-18T21:52:19.727911+0000","last_fullsized":"2024-04-18T21:52:19.727911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"51'10","reported_seq":189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.366709+0000","last_change":"2024-04-18T21:52:18.367906+0000","last_active":"2024-04-18T21:52:18.366709+0000","last_peered":"2024-04-18T21:52:18.366709+0000","last_clean":"2024-04-18T21:52:18.366709+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:18.366709+0000","last_undegraded":"2024-04-18T21:52:18.366709+0000","last_fullsized":"2024-04-18T21:52:18.366709+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:06.245420+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0.0016305250000000001,"stat_sum":{"num_bytes":720896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.726921+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:19.726921+0000","last_peered":"2024-04-18T21:52:19.726921+0000","last_clean":"2024-04-18T21:52:19.726921+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:19.726921+0000","last_undegraded":"2024-04-18T21:52:19.726921+0000","last_fullsized":"2024-04-18T21:52:19.726921+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"59'9","reported_seq":169,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.761070+0000","last_change":"2024-04-18T21:52:15.763319+0000","last_active":"2024-04-18T21:52:15.761070+0000","last_peered":"2024-04-18T21:52:15.761070+0000","last_clean":"2024-04-18T21:52:15.761070+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:15.761070+0000","last_undegraded":"2024-04-18T21:52:15.761070+0000","last_fullsized":"2024-04-18T21:52:15.761070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:25.933067+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.30883592399999998,"stat_sum":{"num_bytes":1605632,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727832+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:19.727832+0000","last_peered":"2024-04-18T21:52:19.727832+0000","last_clean":"2024-04-18T21:52:19.727832+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:19.727832+0000","last_undegraded":"2024-04-18T21:52:19.727832+0000","last_fullsized":"2024-04-18T21:52:19.727832+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":271,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727648+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:19.727648+0000","last_peered":"2024-04-18T21:52:19.727648+0000","last_clean":"2024-04-18T21:52:19.727648+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:19.727648+0000","last_undegraded":"2024-04-18T21:52:19.727648+0000","last_fullsized":"2024-04-18T21:52:19.727648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":176,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827776+0000","last_change":"2024-04-18T21:52:18.469446+0000","last_active":"2024-04-18T21:52:19.827776+0000","last_peered":"2024-04-18T21:52:19.827776+0000","last_clean":"2024-04-18T21:52:19.827776+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:19.827776+0000","last_undegraded":"2024-04-18T21:52:19.827776+0000","last_fullsized":"2024-04-18T21:52:19.827776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:02.621146+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.0021629980000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.9","version":"29'6","reported_seq":189,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.828435+0000","last_change":"2024-04-18T21:52:18.470970+0000","last_active":"2024-04-18T21:52:19.828435+0000","last_peered":"2024-04-18T21:52:19.828435+0000","last_clean":"2024-04-18T21:52:19.828435+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:19.828435+0000","last_undegraded":"2024-04-18T21:52:19.828435+0000","last_fullsized":"2024-04-18T21:52:19.828435+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:06.865747+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.103485668,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.a","version":"51'13","reported_seq":157,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.560980+0000","last_change":"2024-04-18T21:52:15.562372+0000","last_active":"2024-04-18T21:52:15.560980+0000","last_peered":"2024-04-18T21:52:15.560980+0000","last_clean":"2024-04-18T21:52:15.560980+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:15.560980+0000","last_undegraded":"2024-04-18T21:52:15.560980+0000","last_fullsized":"2024-04-18T21:52:15.560980+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.548343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002066048,"stat_sum":{"num_bytes":1146880,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":3268,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"3.b","version":"56'44","reported_seq":183,"reported_epoch":59,"state":"active+clean+snaptrim","last_fresh":"2024-04-18T21:52:15.551477+0000","last_change":"2024-04-18T21:52:15.551556+0000","last_active":"2024-04-18T21:52:15.551477+0000","last_peered":"2024-04-18T21:52:15.551477+0000","last_clean":"2024-04-18T21:52:15.551477+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:15.551477+0000","last_undegraded":"2024-04-18T21:52:15.551477+0000","last_fullsized":"2024-04-18T21:52:15.551477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.007755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.214296292,"stat_sum":{"num_bytes":4833280,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.c","version":"62'51","reported_seq":214,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.729235+0000","last_change":"2024-04-18T21:52:18.375541+0000","last_active":"2024-04-18T21:52:19.729235+0000","last_peered":"2024-04-18T21:52:19.729235+0000","last_clean":"2024-04-18T21:52:19.729235+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:19.729235+0000","last_undegraded":"2024-04-18T21:52:19.729235+0000","last_fullsized":"2024-04-18T21:52:19.729235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:51.557139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036299549999999998,"stat_sum":{"num_bytes":4030464,"num_objects":7,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":24,"num_read_kb":2009,"num_write":61,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.d","version":"0'0","reported_seq":156,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.728375+0000","last_change":"2024-04-18T21:52:18.364025+0000","last_active":"2024-04-18T21:52:19.728375+0000","last_peered":"2024-04-18T21:52:19.728375+0000","last_clean":"2024-04-18T21:52:19.728375+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:19.728375+0000","last_undegraded":"2024-04-18T21:52:19.728375+0000","last_fullsized":"2024-04-18T21:52:19.728375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:57.291644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001192133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.e","version":"62'33","reported_seq":198,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.730397+0000","last_change":"2024-04-18T21:52:18.361111+0000","last_active":"2024-04-18T21:52:19.730397+0000","last_peered":"2024-04-18T21:52:19.730397+0000","last_clean":"2024-04-18T21:52:19.730397+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:19.730397+0000","last_undegraded":"2024-04-18T21:52:19.730397+0000","last_fullsized":"2024-04-18T21:52:19.730397+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.737838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535400000000003,"stat_sum":{"num_bytes":4243456,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":4040,"num_write":21,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.f","version":"62'26","reported_seq":192,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.828469+0000","last_change":"2024-04-18T21:52:18.575622+0000","last_active":"2024-04-18T21:52:19.828469+0000","last_peered":"2024-04-18T21:52:19.828469+0000","last_clean":"2024-04-18T21:52:19.828469+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:19.828469+0000","last_undegraded":"2024-04-18T21:52:19.828469+0000","last_fullsized":"2024-04-18T21:52:19.828469+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:00.439865+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.21226381699999999,"stat_sum":{"num_bytes":4046848,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":2887,"num_write":18,"num_write_kb":4688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.10","version":"59'26","reported_seq":146,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.865053+0000","last_change":"2024-04-18T21:52:15.867407+0000","last_active":"2024-04-18T21:52:15.865053+0000","last_peered":"2024-04-18T21:52:15.865053+0000","last_clean":"2024-04-18T21:52:15.865053+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:15.865053+0000","last_undegraded":"2024-04-18T21:52:15.865053+0000","last_fullsized":"2024-04-18T21:52:15.865053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:18.295879+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0.30822618899999998,"stat_sum":{"num_bytes":3096576,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"3.11","version":"61'30","reported_seq":154,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827816+0000","last_change":"2024-04-18T21:52:18.468562+0000","last_active":"2024-04-18T21:52:19.827816+0000","last_peered":"2024-04-18T21:52:19.827816+0000","last_clean":"2024-04-18T21:52:19.827816+0000","last_became_active":"2024-04-18T21:52:06.594871+0000","last_became_peered":"2024-04-18T21:52:06.594871+0000","last_unstale":"2024-04-18T21:52:19.827816+0000","last_undegraded":"2024-04-18T21:52:19.827816+0000","last_fullsized":"2024-04-18T21:52:19.827816+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'24","last_scrub_stamp":"2024-04-18T21:52:12.154768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:12.154768+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:38.775254+0000","scrub_duration":1591,"objects_trimmed":0,"snaptrim_duration":0.101665902,"stat_sum":{"num_bytes":3899392,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":5761,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,10,1,5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.12","version":"56'11","reported_seq":109,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727669+0000","last_change":"2024-04-18T21:52:18.463510+0000","last_active":"2024-04-18T21:52:19.727669+0000","last_peered":"2024-04-18T21:52:19.727669+0000","last_clean":"2024-04-18T21:52:19.727669+0000","last_became_active":"2024-04-18T21:52:11.882697+0000","last_became_peered":"2024-04-18T21:52:11.882697+0000","last_unstale":"2024-04-18T21:52:19.727669+0000","last_undegraded":"2024-04-18T21:52:19.727669+0000","last_fullsized":"2024-04-18T21:52:19.727669+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'11","last_scrub_stamp":"2024-04-18T21:52:15.060778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:15.060778+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:34.351219+0000","scrub_duration":799,"objects_trimmed":0,"snaptrim_duration":0.079362144999999995,"stat_sum":{"num_bytes":819200,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,10,0,5,6,1],"acting":[9,7,8,10,0,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.13","version":"62'47","reported_seq":150,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.728551+0000","last_change":"2024-04-18T21:52:18.463579+0000","last_active":"2024-04-18T21:52:19.728551+0000","last_peered":"2024-04-18T21:52:19.728551+0000","last_clean":"2024-04-18T21:52:19.728551+0000","last_became_active":"2024-04-18T21:52:16.145636+0000","last_became_peered":"2024-04-18T21:52:16.145636+0000","last_unstale":"2024-04-18T21:52:19.728551+0000","last_undegraded":"2024-04-18T21:52:19.728551+0000","last_fullsized":"2024-04-18T21:52:19.728551+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101046208,"stat_sum":{"num_bytes":3563520,"num_objects":10,"num_object_clones":4,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":25,"num_read_kb":3175,"num_write":22,"num_write_kb":4376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4,8,6,11,1,9],"acting":[7,5,4,8,6,11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":53977088,"num_objects":73,"num_object_clones":23,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":4,"num_read":133,"num_read_kb":27106,"num_write":170,"num_write_kb":33224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":105717760,"data_stored":118456320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":507,"ondisk_log_size":507,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378637,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38472,"kb_used_data":7288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333368,"statfs":{"total":96636764160,"available":96597368832,"internally_reserved":0,"allocated":7462912,"data_stored":8772460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378637,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41668,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330172,"statfs":{"total":96636764160,"available":96594096128,"internally_reserved":0,"allocated":10739712,"data_stored":12167787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378638,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":10188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":10432512,"data_stored":11512684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":12172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":12464128,"data_stored":13609836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378638,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39672,"kb_used_data":8488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332168,"statfs":{"total":96636764160,"available":96596140032,"internally_reserved":0,"allocated":8691712,"data_stored":9150007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411341,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33348,"kb_used_data":6324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":6475776,"data_stored":6663735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411341,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35484,"kb_used_data":8460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336356,"statfs":{"total":96636764160,"available":96600428544,"internally_reserved":0,"allocated":8663040,"data_stored":9760855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411341,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42436,"kb_used_data":11256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329404,"statfs":{"total":96636764160,"available":96593309696,"internally_reserved":0,"allocated":11526144,"data_stored":12136168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411342,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42848,"kb_used_data":11664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328992,"statfs":{"total":96636764160,"available":96592887808,"internally_reserved":0,"allocated":11943936,"data_stored":12746295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411340,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35624,"kb_used_data":8600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336216,"statfs":{"total":96636764160,"available":96600285184,"internally_reserved":0,"allocated":8806400,"data_stored":9367639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444045,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35688,"kb_used_data":8664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336152,"statfs":{"total":96636764160,"available":96600219648,"internally_reserved":0,"allocated":8871936,"data_stored":9498167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444045,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34876,"kb_used_data":7852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336964,"statfs":{"total":96636764160,"available":96601051136,"internally_reserved":0,"allocated":8040448,"data_stored":8805943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7766016,"data_stored":8622080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11800576,"data_stored":13225984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11608064,"data_stored":13029376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7991296,"data_stored":8740864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9768960,"data_stored":11128832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10813440,"data_stored":11718656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":10420224,"data_stored":11788288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7430144,"data_stored":8822784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":7372800,"data_stored":7868416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5775360,"data_stored":6254592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6799360,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8171520,"data_stored":8867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:20.456 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-18T21:52:20.456 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:52:20.456 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-18T21:52:20.456 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 12 2024-04-18T21:52:20.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:20.467 INFO:tasks.rados.rados.0.smithi097.stdout:193: done (1 left) 2024-04-18T21:52:20.467 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 6 current snap is 15 2024-04-18T21:52:20.468 INFO:tasks.rados.rados.0.smithi097.stdout:194: seq_num 87 ranges {0=704512,704512=688128,1392640=81920} 2024-04-18T21:52:20.471 INFO:tasks.rados.rados.0.smithi097.stdout:194: writing smithi09731216-6 from 0 to 704512 tid 1 2024-04-18T21:52:20.475 INFO:tasks.rados.rados.0.smithi097.stdout:194: writing smithi09731216-6 from 704512 to 1392640 tid 2 2024-04-18T21:52:20.476 INFO:tasks.rados.rados.0.smithi097.stdout:194: writing smithi09731216-6 from 1392640 to 1474560 tid 3 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 20 current snap is 15 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:191: finishing write tid 5 to smithi09731216-28 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:191: oid 28 updating version 0 to 33 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:191: oid 28 version 33 is already newer than 31 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:191: oid 28 version 33 is already newer than 30 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:191: oid 28 version 33 is already newer than 32 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 28 v 33 (ObjNum 86 snap 14 seq_num 86) dirty exists 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:191: left oid 28 (ObjNum 86 snap 14 seq_num 86) 2024-04-18T21:52:20.477 INFO:tasks.rados.rados.0.smithi097.stdout:195: seq_num 88 ranges {1327104=720896,2048000=442368,2490368=327680} 2024-04-18T21:52:20.488 INFO:tasks.rados.rados.0.smithi097.stdout:195: writing smithi09731216-20 from 1327104 to 2048000 tid 1 2024-04-18T21:52:20.491 INFO:tasks.rados.rados.0.smithi097.stdout:195: writing smithi09731216-20 from 2048000 to 2490368 tid 2 2024-04-18T21:52:20.493 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:20.483+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:20.493 INFO:tasks.rados.rados.0.smithi097.stdout:195: writing smithi09731216-20 from 2490368 to 2818048 tid 3 2024-04-18T21:52:20.493 INFO:tasks.rados.rados.0.smithi097.stdout:191: done (2 left) 2024-04-18T21:52:20.493 INFO:tasks.rados.rados.0.smithi097.stdout:196: setattr oid 50 current snap is 15 2024-04-18T21:52:20.494 INFO:tasks.rados.rados.0.smithi097.stdout:197: read oid 31 snap 4 2024-04-18T21:52:20.494 INFO:tasks.rados.rados.0.smithi097.stdout:197: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-18T21:52:20.494 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 19 current snap is 15 2024-04-18T21:52:20.494 INFO:tasks.rados.rados.0.smithi097.stdout:198: seq_num 89 ranges {1327104=737280,2064384=573440,2637824=196608} 2024-04-18T21:52:20.506 INFO:tasks.rados.rados.0.smithi097.stdout:198: writing smithi09731216-19 from 1327104 to 2064384 tid 1 2024-04-18T21:52:20.509 INFO:tasks.rados.rados.0.smithi097.stdout:198: writing smithi09731216-19 from 2064384 to 2637824 tid 2 2024-04-18T21:52:20.510 INFO:tasks.rados.rados.0.smithi097.stdout:198: writing smithi09731216-19 from 2637824 to 2834432 tid 3 2024-04-18T21:52:20.510 INFO:tasks.rados.rados.0.smithi097.stdout:199: snap_create 2024-04-18T21:52:20.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:20.536 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:20.537 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:20.537 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:20.537 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:20.537 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:20.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:20.587 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:20.583+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:20.655 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:20.656 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:97 3.4s0 3:209f2930:::smithi09731216-33:head [append 0~540672,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.679062463000001, 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4026191889999999, 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985858+0000", 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.92e-06 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987540+0000", 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001681463 2024-04-18T21:52:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987566+0000", 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6043000000000001e-05 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987586+0000", 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9930000000000001e-05 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087950+0000", 2024-04-18T21:52:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100363824 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087956+0000", 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9200000000000001e-06 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388374+0000", 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3004186340000001 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388475+0000", 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100455 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:98 3.4s0 3:209f2930:::smithi09731216-33:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.677301061000001, 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.401002842, 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 98 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987618+0000", 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5e-06 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987625+0000", 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6769999999999998e-06 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987637+0000", 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1693e-05 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987646+0000", 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3700000000000001e-06 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088140+0000", 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10049390900000001 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088145+0000", 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3600000000000004e-06 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388530+0000", 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300384269 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388620+0000", 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0063999999999997e-05 2024-04-18T21:52:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:99 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.677258320999996, 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503241456, 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987661+0000", 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5999999999999991e-07 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987664+0000", 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6459999999999999e-06 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987675+0000", 2024-04-18T21:52:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0967e-05 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987684+0000", 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6929999999999996e-06 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088241+0000", 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100557049 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088244+0000", 2024-04-18T21:52:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2729999999999998e-06 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388746+0000", 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300502061 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388878+0000", 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131533 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490901+0000", 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102023274 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:117 3.bs0 3:d4cd3926:::smithi09731216-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.477189312, 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.520974363, 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 117 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187730+0000", 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.127e-06 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187735+0000", 2024-04-18T21:52:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.536e-06 2024-04-18T21:52:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187752+0000", 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6186999999999999e-05 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187762+0000", 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0587e-05 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281120+0000", 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093357525999999996 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281296+0000", 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017659300000000001 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405205+0000", 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.123908465 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405449+0000", 2024-04-18T21:52:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000244505 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708703+0000", 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30325383700000003 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:126 3.14s0 3:2c76b789:::smithi09731216-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.1586608, 2024-04-18T21:52:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50118609800000002, 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 126 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506258+0000", 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506267+0000", 2024-04-18T21:52:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7159999999999992e-06 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506287+0000", 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9542999999999999e-05 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506301+0000", 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4572999999999999e-05 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511483+0000", 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051813989999999997 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511616+0000", 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013376499999999999 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803750+0000", 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29213323699999999 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803998+0000", 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024861799999999998 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.007443+0000", 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20344494699999999 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.16s4 32/31 ECSubWrite(tid=15, reqid=client.4509.0:110, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 31.576761685000001, 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.42111731499999999, 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088157+0000", 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088207+0000", 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0584999999999998e-05 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088241+0000", 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3784e-05 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088269+0000", 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8314999999999999e-05 2024-04-18T21:52:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408130+0000", 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31986045299999999 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408140+0000", 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0248000000000001e-05 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509225+0000", 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10108521500000001 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509274+0000", 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8374999999999997e-05 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:216 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.649005304999999, 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51394664400000001, 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015916+0000", 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0130000000000002e-06 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015930+0000", 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4705000000000001e-05 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015964+0000", 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4072000000000002e-05 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015996+0000", 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1198999999999997e-05 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016142+0000", 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014630900000000001 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016155+0000", 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2581e-05 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.020038+0000", 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038838789999999998 2024-04-18T21:52:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.020191+0000", 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000152559 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.021481+0000", 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012896680000000001 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128155+0000", 2024-04-18T21:52:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106673879 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.529681+0000", 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40152656199999998 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.529859+0000", 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017821800000000001 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:217 3.bs0 3:d90895f4:::smithi09731216-9:head [rollback 2] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.648868271000001, 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51416292299999999, 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217 2024-04-18T21:52:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016051+0000", 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6470000000000001e-06 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016062+0000", 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0451999999999999e-05 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016085+0000", 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2776e-05 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016103+0000", 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8131e-05 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016278+0000", 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017520200000000001 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016289+0000", 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1048e-05 2024-04-18T21:52:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.023057+0000", 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067683750000000001 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.023334+0000", 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027651500000000002 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.024976+0000", 2024-04-18T21:52:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016423379999999999 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128157+0000", 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10318049899999999 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.530073+0000", 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40191629299999998 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.530213+0000", 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013964700000000001 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:218 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.648784966000001, 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61382267800000001, 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 218 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016134+0000", 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016141+0000", 2024-04-18T21:52:20.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5719999999999999e-06 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016161+0000", 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9814e-05 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016177+0000", 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6028e-05 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016365+0000", 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018808400000000001 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016384+0000", 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-05 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026326+0000", 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0099418909999999996 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026591+0000", 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026506899999999999 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.027603+0000", 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010121799999999999 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128223+0000", 2024-04-18T21:52:20.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100619874 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629817+0000", 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50159395299999998 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629956+0000", 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013901300000000001 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.717 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:298 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~720896] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 22.603447422999999, 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.422019223, 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.718 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 298 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061474+0000", 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2770000000000001e-06 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063120+0000", 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001645903 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063156+0000", 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.642e-05 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063174+0000", 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7621e-05 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063316+0000", 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000141753 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063330+0000", 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4324999999999999e-05 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.166460+0000", 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103130208 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.166636+0000", 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175792 2024-04-18T21:52:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.170069+0000", 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003432778 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279315+0000", 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.109246564 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.483316+0000", 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20400066 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.483490+0000", 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017392199999999999 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:299 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 22.600603241000002, 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.518657543, 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 299 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064317+0000", 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5900000000000002e-06 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065148+0000", 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000830709 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065184+0000", 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5738999999999998e-05 2024-04-18T21:52:20.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065209+0000", 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5627e-05 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065508+0000", 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029824799999999998 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065539+0000", 2024-04-18T21:52:20.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0871999999999999e-05 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.172826+0000", 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.107287676 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.174217+0000", 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001390891 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.177064+0000", 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028466120000000001 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279319+0000", 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102254998 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.582844+0000", 2024-04-18T21:52:20.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30352516899999998 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.582972+0000", 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000128412 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:300 3.bs0 3:d90895f4:::smithi09731216-9:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:20.731 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 22.599658647999998, 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.71857623199999998, 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 300 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065261+0000", 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9690000000000001e-06 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065272+0000", 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0545e-05 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065299+0000", 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6927000000000002e-05 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065319+0000", 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0035e-05 2024-04-18T21:52:20.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165723+0000", 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100404099 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165747+0000", 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.438e-05 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279995+0000", 2024-04-18T21:52:20.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.114247892 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280276+0000", 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028071299999999998 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281155+0000", 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087942000000000001 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.381976+0000", 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100820639 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783683+0000", 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40170656199999999 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783836+0000", 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000153051 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.737 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:301 3.bs0 3:d90895f4:::smithi09731216-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 22.599526493999999, 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.82307189599999997, 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.738 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 301 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065393+0000", 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.553e-06 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065401+0000", 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9719999999999992e-06 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065425+0000", 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3547e-05 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065447+0000", 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2234000000000001e-05 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165843+0000", 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100395994 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165860+0000", 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6871000000000002e-05 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281656+0000", 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11579645099999999 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281793+0000", 2024-04-18T21:52:20.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013725 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783958+0000", 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50216431100000003 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.784112+0000", 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015411799999999999 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.888463+0000", 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10435159500000001 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:422 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~294912] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.2180380419999999, 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61132010699999995, 2024-04-18T21:52:20.744 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 422 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446882+0000", 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5459999999999998e-06 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447318+0000", 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00043564599999999998 2024-04-18T21:52:20.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447349+0000", 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0888e-05 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447365+0000", 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5593000000000001e-05 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447499+0000", 2024-04-18T21:52:20.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013438999999999999 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447509+0000", 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6509999999999996e-06 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.556315+0000", 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10880680299999999 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.556603+0000", 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028725400000000002 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.558288+0000", 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001684786 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638308+0000", 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.080020547999999997 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058105+0000", 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41979719399999998 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058200+0000", 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4808000000000004e-05 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.750 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:423 3.2s0 3:44bf1f18:::smithi09731216-13:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.2175208990000002, 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61117605200000003, 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 423 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447398+0000", 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.353e-06 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447416+0000", 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7353000000000001e-05 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447431+0000", 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5645e-05 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447441+0000", 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2690000000000005e-06 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447569+0000", 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012837999999999999 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447573+0000", 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3270000000000002e-06 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.559335+0000", 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11176207000000001 2024-04-18T21:52:20.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.559597+0000", 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026142 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.560692+0000", 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010952799999999999 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638310+0000", 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.077617829999999999 2024-04-18T21:52:20.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058483+0000", 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.42017316199999999 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058573+0000", 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9963000000000006e-05 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:424 3.2s0 3:44bf1f18:::smithi09731216-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.2174228060000001, 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.81405277300000001, 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 424 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447496+0000", 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1699999999999997e-07 2024-04-18T21:52:20.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447501+0000", 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5059999999999997e-06 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447513+0000", 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2592000000000001e-05 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447526+0000", 2024-04-18T21:52:20.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2309e-05 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.547672+0000", 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10014685800000001 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.547684+0000", 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1232e-05 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.756870+0000", 2024-04-18T21:52:20.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.209186237 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.756965+0000", 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5280999999999998e-05 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.059024+0000", 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30205871899999998 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.059128+0000", 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010387600000000001 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.261548+0000", 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.202420246 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:434 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:20.762 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.2173744299999996, 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.58875951599999998, 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 434 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999998e-07 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447549+0000", 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7740000000000004e-06 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447560+0000", 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0522e-05 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447569+0000", 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.312e-06 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447614+0000", 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5370000000000001e-05 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447620+0000", 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.502e-06 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638433+0000", 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.190813609 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638582+0000", 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014889800000000001 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.036303+0000", 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39772072600000002 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:435 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.2173280069999999, 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.58911970899999999, 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 435 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447591+0000", 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2270000000000001e-06 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447598+0000", 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6630000000000004e-06 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447616+0000", 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8587999999999999e-05 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447631+0000", 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4326e-05 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447673+0000", 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2159000000000001e-05 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447677+0000", 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0799999999999999e-06 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639051+0000", 2024-04-18T21:52:20.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19137447299999999 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639173+0000", 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121443 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.036710+0000", 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39753674999999999 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:436 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.2172657649999996, 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.59017562599999995, 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 436 2024-04-18T21:52:20.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447653+0000", 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1540000000000001e-06 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447661+0000", 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7449999999999995e-06 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447679+0000", 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8263000000000001e-05 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447693+0000", 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3642000000000001e-05 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447729+0000", 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6124000000000002e-05 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447733+0000", 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9620000000000004e-06 2024-04-18T21:52:20.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639451+0000", 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19171812099999999 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639522+0000", 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0956000000000003e-05 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.037828+0000", 2024-04-18T21:52:20.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39830565899999998 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(osd.1.19:2 3.bs0 3:d688e784:::smithi09731216-24:head [assert-version v39,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.518729+0000", 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.1461886330000004, 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41734706799999999, 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:20.778 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518729+0000", 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518729+0000", 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518730+0000", 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.094e-06 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518735+0000", 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.617e-06 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518744+0000", 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2709999999999998e-06 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518753+0000", 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7090000000000008e-06 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518874+0000", 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012108399999999999 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518889+0000", 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4636e-05 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.538447+0000", 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019558351000000002 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.538594+0000", 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014643600000000001 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.936076+0000", 2024-04-18T21:52:20.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39748286999999999 2024-04-18T21:52:20.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.783 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.783 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.783 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:20.784 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:20.784 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:20.785 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:20.785 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:20.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:20.786 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:20.786 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:20.787 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:20.789 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-04-18T21:52:20.470345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54567456,"num_objects":75,"num_object_clones":23,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":4,"num_read":259,"num_read_kb":27215,"num_write":351,"num_write_kb":36162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":644,"ondisk_log_size":644,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":234,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":464844,"kb_used_data":111444,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1131997236,"statfs":{"total":1159641169920,"available":1159165169664,"internally_reserved":0,"allocated":114118656,"data_stored":124191576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,0,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2777088,"num_objects":2,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":6,"num_read_kb":5763,"num_write":9,"num_write_kb":1912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025521"},"pg_stats":[{"pgid":"3.19","version":"56'22","reported_seq":208,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827516+0000","last_change":"2024-04-18T21:52:18.468380+0000","last_active":"2024-04-18T21:52:19.827516+0000","last_peered":"2024-04-18T21:52:19.827516+0000","last_clean":"2024-04-18T21:52:19.827516+0000","last_became_active":"2024-04-18T21:51:48.694394+0000","last_became_peered":"2024-04-18T21:51:48.694394+0000","last_unstale":"2024-04-18T21:52:19.827516+0000","last_undegraded":"2024-04-18T21:52:19.827516+0000","last_fullsized":"2024-04-18T21:52:19.827516+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'9","last_scrub_stamp":"2024-04-18T21:52:01.766722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.766722+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:21.638148+0000","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0.101186106,"stat_sum":{"num_bytes":1376256,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.18","version":"51'20","reported_seq":201,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827463+0000","last_change":"2024-04-18T21:52:18.372220+0000","last_active":"2024-04-18T21:52:19.827463+0000","last_peered":"2024-04-18T21:52:19.827463+0000","last_clean":"2024-04-18T21:52:19.827463+0000","last_became_active":"2024-04-18T21:51:48.590342+0000","last_became_peered":"2024-04-18T21:51:48.590342+0000","last_unstale":"2024-04-18T21:52:19.827463+0000","last_undegraded":"2024-04-18T21:52:19.827463+0000","last_fullsized":"2024-04-18T21:52:19.827463+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"42'16","last_scrub_stamp":"2024-04-18T21:52:00.260693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.260693+0000","objects_scrubbed":4,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.945044+0000","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0.0053765150000000001,"stat_sum":{"num_bytes":2523136,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2258,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.17","version":"51'8","reported_seq":182,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.376323+0000","last_change":"2024-04-18T21:52:18.477294+0000","last_active":"2024-04-18T21:52:18.376323+0000","last_peered":"2024-04-18T21:52:18.376323+0000","last_clean":"2024-04-18T21:52:18.376323+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:51:48.483038+0000","last_unstale":"2024-04-18T21:52:18.376323+0000","last_undegraded":"2024-04-18T21:52:18.376323+0000","last_fullsized":"2024-04-18T21:52:18.376323+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:13.107163+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.10287663,"stat_sum":{"num_bytes":671744,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.16","version":"32'3","reported_seq":181,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.366807+0000","last_change":"2024-04-18T21:52:18.368049+0000","last_active":"2024-04-18T21:52:18.366807+0000","last_peered":"2024-04-18T21:52:18.366807+0000","last_clean":"2024-04-18T21:52:18.366807+0000","last_became_active":"2024-04-18T21:51:48.984342+0000","last_became_peered":"2024-04-18T21:51:48.984342+0000","last_unstale":"2024-04-18T21:52:18.366807+0000","last_undegraded":"2024-04-18T21:52:18.366807+0000","last_fullsized":"2024-04-18T21:52:18.366807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:06.462212+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.0015130409999999999,"stat_sum":{"num_bytes":606208,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.15","version":"51'9","reported_seq":187,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727434+0000","last_change":"2024-04-18T21:52:18.568579+0000","last_active":"2024-04-18T21:52:19.727434+0000","last_peered":"2024-04-18T21:52:19.727434+0000","last_clean":"2024-04-18T21:52:19.727434+0000","last_became_active":"2024-04-18T21:51:48.884244+0000","last_became_peered":"2024-04-18T21:51:48.884244+0000","last_unstale":"2024-04-18T21:52:19.727434+0000","last_undegraded":"2024-04-18T21:52:19.727434+0000","last_fullsized":"2024-04-18T21:52:19.727434+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.274776+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.00073016800000000001,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.14","version":"51'8","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.455291+0000","last_change":"2024-04-18T21:52:15.456411+0000","last_active":"2024-04-18T21:52:15.455291+0000","last_peered":"2024-04-18T21:52:15.455291+0000","last_clean":"2024-04-18T21:52:15.455291+0000","last_became_active":"2024-04-18T21:51:48.285813+0000","last_became_peered":"2024-04-18T21:51:48.285813+0000","last_unstale":"2024-04-18T21:52:15.455291+0000","last_undegraded":"2024-04-18T21:52:15.455291+0000","last_fullsized":"2024-04-18T21:52:15.455291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.605862+0000","scrub_duration":1515,"objects_trimmed":0,"snaptrim_duration":0.0017272139999999999,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.459239+0000","last_change":"2024-04-18T21:51:33.008394+0000","last_active":"2024-04-18T21:52:15.459239+0000","last_peered":"2024-04-18T21:52:15.459239+0000","last_clean":"2024-04-18T21:52:15.459239+0000","last_became_active":"2024-04-18T21:51:32.907615+0000","last_became_peered":"2024-04-18T21:51:32.907615+0000","last_unstale":"2024-04-18T21:52:15.459239+0000","last_undegraded":"2024-04-18T21:52:15.459239+0000","last_fullsized":"2024-04-18T21:52:15.459239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:37.931928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'6","reported_seq":180,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.781520+0000","last_change":"2024-04-18T21:52:18.780498+0000","last_active":"2024-04-18T21:52:18.781520+0000","last_peered":"2024-04-18T21:52:18.781520+0000","last_clean":"2024-04-18T21:52:18.781520+0000","last_became_active":"2024-04-18T21:51:48.480072+0000","last_became_peered":"2024-04-18T21:51:48.480072+0000","last_unstale":"2024-04-18T21:52:18.781520+0000","last_undegraded":"2024-04-18T21:52:18.781520+0000","last_fullsized":"2024-04-18T21:52:18.781520+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.144353+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.413638432,"stat_sum":{"num_bytes":589824,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,7,10,6,1,5],"acting":[11,3,4,7,10,6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":91,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.729008+0000","last_change":"2024-04-18T21:51:32.922907+0000","last_active":"2024-04-18T21:52:19.729008+0000","last_peered":"2024-04-18T21:52:19.729008+0000","last_clean":"2024-04-18T21:52:19.729008+0000","last_became_active":"2024-04-18T21:51:32.922058+0000","last_became_peered":"2024-04-18T21:51:32.922058+0000","last_unstale":"2024-04-18T21:52:19.729008+0000","last_undegraded":"2024-04-18T21:52:19.729008+0000","last_fullsized":"2024-04-18T21:52:19.729008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.082752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":171,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.476485+0000","last_change":"2024-04-18T21:52:18.572183+0000","last_active":"2024-04-18T21:52:18.476485+0000","last_peered":"2024-04-18T21:52:18.476485+0000","last_clean":"2024-04-18T21:52:18.476485+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:51:48.380296+0000","last_unstale":"2024-04-18T21:52:18.476485+0000","last_undegraded":"2024-04-18T21:52:18.476485+0000","last_fullsized":"2024-04-18T21:52:18.476485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:51.625973+0000","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.096017020999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7,10,9,1,4,5],"acting":[8,3,7,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.928705+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:19.928705+0000","last_peered":"2024-04-18T21:52:19.928705+0000","last_clean":"2024-04-18T21:52:19.928705+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:19.928705+0000","last_undegraded":"2024-04-18T21:52:19.928705+0000","last_fullsized":"2024-04-18T21:52:19.928705+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:38.077632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.456707+0000","last_change":"2024-04-18T21:52:15.458162+0000","last_active":"2024-04-18T21:52:15.456707+0000","last_peered":"2024-04-18T21:52:15.456707+0000","last_clean":"2024-04-18T21:52:15.456707+0000","last_became_active":"2024-04-18T21:51:48.186532+0000","last_became_peered":"2024-04-18T21:51:48.186532+0000","last_unstale":"2024-04-18T21:52:15.456707+0000","last_undegraded":"2024-04-18T21:52:15.456707+0000","last_fullsized":"2024-04-18T21:52:15.456707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.146084+0000","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.0033094909999999999,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3170,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,0,9,11],"acting":[10,2,1,5,8,0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.366253+0000","last_change":"2024-04-18T21:51:33.023283+0000","last_active":"2024-04-18T21:52:18.366253+0000","last_peered":"2024-04-18T21:52:18.366253+0000","last_clean":"2024-04-18T21:52:18.366253+0000","last_became_active":"2024-04-18T21:51:33.022420+0000","last_became_peered":"2024-04-18T21:51:33.022420+0000","last_unstale":"2024-04-18T21:52:18.366253+0000","last_undegraded":"2024-04-18T21:52:18.366253+0000","last_fullsized":"2024-04-18T21:52:18.366253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:42.329927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"61'15","reported_seq":192,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827756+0000","last_change":"2024-04-18T21:52:18.369180+0000","last_active":"2024-04-18T21:52:19.827756+0000","last_peered":"2024-04-18T21:52:19.827756+0000","last_clean":"2024-04-18T21:52:19.827756+0000","last_became_active":"2024-04-18T21:51:48.575779+0000","last_became_peered":"2024-04-18T21:51:48.575779+0000","last_unstale":"2024-04-18T21:52:19.827756+0000","last_undegraded":"2024-04-18T21:52:19.827756+0000","last_fullsized":"2024-04-18T21:52:19.827756+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.420167+0000","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.001025174,"stat_sum":{"num_bytes":868352,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,7,11,3,0],"acting":[5,8,10,6,7,11,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.3","version":"59'84","reported_seq":257,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.763076+0000","last_change":"2024-04-18T21:52:15.765341+0000","last_active":"2024-04-18T21:52:15.763076+0000","last_peered":"2024-04-18T21:52:15.763076+0000","last_clean":"2024-04-18T21:52:15.763076+0000","last_became_active":"2024-04-18T21:51:48.279953+0000","last_became_peered":"2024-04-18T21:51:48.279953+0000","last_unstale":"2024-04-18T21:52:15.763076+0000","last_undegraded":"2024-04-18T21:52:15.763076+0000","last_fullsized":"2024-04-18T21:52:15.763076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:24.270053+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.28746011199999999,"stat_sum":{"num_bytes":10403840,"num_objects":11,"num_object_clones":4,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11,6,1,5,2,0],"acting":[4,3,11,6,1,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727403+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:19.727403+0000","last_peered":"2024-04-18T21:52:19.727403+0000","last_clean":"2024-04-18T21:52:19.727403+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:19.727403+0000","last_undegraded":"2024-04-18T21:52:19.727403+0000","last_fullsized":"2024-04-18T21:52:19.727403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"59'22","reported_seq":189,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.957919+0000","last_change":"2024-04-18T21:52:15.959178+0000","last_active":"2024-04-18T21:52:15.957919+0000","last_peered":"2024-04-18T21:52:15.957919+0000","last_clean":"2024-04-18T21:52:15.957919+0000","last_became_active":"2024-04-18T21:51:48.481064+0000","last_became_peered":"2024-04-18T21:51:48.481064+0000","last_unstale":"2024-04-18T21:52:15.957919+0000","last_undegraded":"2024-04-18T21:52:15.957919+0000","last_fullsized":"2024-04-18T21:52:15.957919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:15.113096+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.50073931699999996,"stat_sum":{"num_bytes":3866624,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727911+0000","last_change":"2024-04-18T21:51:32.917034+0000","last_active":"2024-04-18T21:52:19.727911+0000","last_peered":"2024-04-18T21:52:19.727911+0000","last_clean":"2024-04-18T21:52:19.727911+0000","last_became_active":"2024-04-18T21:51:32.916161+0000","last_became_peered":"2024-04-18T21:51:32.916161+0000","last_unstale":"2024-04-18T21:52:19.727911+0000","last_undegraded":"2024-04-18T21:52:19.727911+0000","last_fullsized":"2024-04-18T21:52:19.727911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.463251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"51'10","reported_seq":189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-18T21:52:18.366709+0000","last_change":"2024-04-18T21:52:18.367906+0000","last_active":"2024-04-18T21:52:18.366709+0000","last_peered":"2024-04-18T21:52:18.366709+0000","last_clean":"2024-04-18T21:52:18.366709+0000","last_became_active":"2024-04-18T21:51:48.386337+0000","last_became_peered":"2024-04-18T21:51:48.386337+0000","last_unstale":"2024-04-18T21:52:18.366709+0000","last_undegraded":"2024-04-18T21:52:18.366709+0000","last_fullsized":"2024-04-18T21:52:18.366709+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:06.245420+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0.0016305250000000001,"stat_sum":{"num_bytes":720896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":1,"num_write_kb":528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.726921+0000","last_change":"2024-04-18T21:51:33.042110+0000","last_active":"2024-04-18T21:52:19.726921+0000","last_peered":"2024-04-18T21:52:19.726921+0000","last_clean":"2024-04-18T21:52:19.726921+0000","last_became_active":"2024-04-18T21:51:33.041631+0000","last_became_peered":"2024-04-18T21:51:33.041631+0000","last_unstale":"2024-04-18T21:52:19.726921+0000","last_undegraded":"2024-04-18T21:52:19.726921+0000","last_fullsized":"2024-04-18T21:52:19.726921+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.305192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"59'9","reported_seq":169,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.761070+0000","last_change":"2024-04-18T21:52:15.763319+0000","last_active":"2024-04-18T21:52:15.761070+0000","last_peered":"2024-04-18T21:52:15.761070+0000","last_clean":"2024-04-18T21:52:15.761070+0000","last_became_active":"2024-04-18T21:51:48.286561+0000","last_became_peered":"2024-04-18T21:51:48.286561+0000","last_unstale":"2024-04-18T21:52:15.761070+0000","last_undegraded":"2024-04-18T21:52:15.761070+0000","last_fullsized":"2024-04-18T21:52:15.761070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:25.933067+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.30883592399999998,"stat_sum":{"num_bytes":1605632,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727832+0000","last_change":"2024-04-18T21:51:32.919412+0000","last_active":"2024-04-18T21:52:19.727832+0000","last_peered":"2024-04-18T21:52:19.727832+0000","last_clean":"2024-04-18T21:52:19.727832+0000","last_became_active":"2024-04-18T21:51:32.918826+0000","last_became_peered":"2024-04-18T21:51:32.918826+0000","last_unstale":"2024-04-18T21:52:19.727832+0000","last_undegraded":"2024-04-18T21:52:19.727832+0000","last_fullsized":"2024-04-18T21:52:19.727832+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:31.945729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":271,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727648+0000","last_change":"2024-04-18T21:51:30.083293+0000","last_active":"2024-04-18T21:52:19.727648+0000","last_peered":"2024-04-18T21:52:19.727648+0000","last_clean":"2024-04-18T21:52:19.727648+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:19.727648+0000","last_undegraded":"2024-04-18T21:52:19.727648+0000","last_fullsized":"2024-04-18T21:52:19.727648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:34.099278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'2","reported_seq":176,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827776+0000","last_change":"2024-04-18T21:52:18.469446+0000","last_active":"2024-04-18T21:52:19.827776+0000","last_peered":"2024-04-18T21:52:19.827776+0000","last_clean":"2024-04-18T21:52:19.827776+0000","last_became_active":"2024-04-18T21:51:48.376186+0000","last_became_peered":"2024-04-18T21:51:48.376186+0000","last_unstale":"2024-04-18T21:52:19.827776+0000","last_undegraded":"2024-04-18T21:52:19.827776+0000","last_fullsized":"2024-04-18T21:52:19.827776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:02.621146+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.0021629980000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,0,1],"acting":[3,10,2,4,9,5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.9","version":"29'6","reported_seq":189,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.828435+0000","last_change":"2024-04-18T21:52:18.470970+0000","last_active":"2024-04-18T21:52:19.828435+0000","last_peered":"2024-04-18T21:52:19.828435+0000","last_clean":"2024-04-18T21:52:19.828435+0000","last_became_active":"2024-04-18T21:51:48.382852+0000","last_became_peered":"2024-04-18T21:51:48.382852+0000","last_unstale":"2024-04-18T21:52:19.828435+0000","last_undegraded":"2024-04-18T21:52:19.828435+0000","last_fullsized":"2024-04-18T21:52:19.828435+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:06.865747+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.103485668,"stat_sum":{"num_bytes":32768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,1,5,9,4,6],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.a","version":"51'13","reported_seq":157,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.560980+0000","last_change":"2024-04-18T21:52:15.562372+0000","last_active":"2024-04-18T21:52:15.560980+0000","last_peered":"2024-04-18T21:52:15.560980+0000","last_clean":"2024-04-18T21:52:15.560980+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:15.560980+0000","last_undegraded":"2024-04-18T21:52:15.560980+0000","last_fullsized":"2024-04-18T21:52:15.560980+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:01.548343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002066048,"stat_sum":{"num_bytes":1146880,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":3268,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,7],"acting":[6,4,9,2,3,1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"3.b","version":"56'44","reported_seq":183,"reported_epoch":59,"state":"active+clean+snaptrim","last_fresh":"2024-04-18T21:52:15.551477+0000","last_change":"2024-04-18T21:52:15.551556+0000","last_active":"2024-04-18T21:52:15.551477+0000","last_peered":"2024-04-18T21:52:15.551477+0000","last_clean":"2024-04-18T21:52:15.551477+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:15.551477+0000","last_undegraded":"2024-04-18T21:52:15.551477+0000","last_fullsized":"2024-04-18T21:52:15.551477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:55.007755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.214296292,"stat_sum":{"num_bytes":4833280,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,2,1,7,8,6],"acting":[10,3,4,2,1,7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.c","version":"62'51","reported_seq":214,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.729235+0000","last_change":"2024-04-18T21:52:18.375541+0000","last_active":"2024-04-18T21:52:19.729235+0000","last_peered":"2024-04-18T21:52:19.729235+0000","last_clean":"2024-04-18T21:52:19.729235+0000","last_became_active":"2024-04-18T21:51:43.067323+0000","last_became_peered":"2024-04-18T21:51:43.067323+0000","last_unstale":"2024-04-18T21:52:19.729235+0000","last_undegraded":"2024-04-18T21:52:19.729235+0000","last_fullsized":"2024-04-18T21:52:19.729235+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:51.557139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036299549999999998,"stat_sum":{"num_bytes":4030464,"num_objects":7,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":24,"num_read_kb":2009,"num_write":61,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,0,10,9,2],"acting":[5,8,6,1,0,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.d","version":"0'0","reported_seq":156,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.728375+0000","last_change":"2024-04-18T21:52:18.364025+0000","last_active":"2024-04-18T21:52:19.728375+0000","last_peered":"2024-04-18T21:52:19.728375+0000","last_clean":"2024-04-18T21:52:19.728375+0000","last_became_active":"2024-04-18T21:51:43.127692+0000","last_became_peered":"2024-04-18T21:51:43.127692+0000","last_unstale":"2024-04-18T21:52:19.728375+0000","last_undegraded":"2024-04-18T21:52:19.728375+0000","last_fullsized":"2024-04-18T21:52:19.728375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:57.291644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001192133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,6,3,8,4,2],"acting":[7,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.e","version":"62'33","reported_seq":198,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.730397+0000","last_change":"2024-04-18T21:52:18.361111+0000","last_active":"2024-04-18T21:52:19.730397+0000","last_peered":"2024-04-18T21:52:19.730397+0000","last_clean":"2024-04-18T21:52:19.730397+0000","last_became_active":"2024-04-18T21:51:43.166005+0000","last_became_peered":"2024-04-18T21:51:43.166005+0000","last_unstale":"2024-04-18T21:52:19.730397+0000","last_undegraded":"2024-04-18T21:52:19.730397+0000","last_fullsized":"2024-04-18T21:52:19.730397+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:09.737838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535400000000003,"stat_sum":{"num_bytes":4243456,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":4040,"num_write":21,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,4,8,11,9,7],"acting":[2,0,1,4,8,11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.f","version":"62'26","reported_seq":192,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.828469+0000","last_change":"2024-04-18T21:52:18.575622+0000","last_active":"2024-04-18T21:52:19.828469+0000","last_peered":"2024-04-18T21:52:19.828469+0000","last_clean":"2024-04-18T21:52:19.828469+0000","last_became_active":"2024-04-18T21:51:43.242992+0000","last_became_peered":"2024-04-18T21:51:43.242992+0000","last_unstale":"2024-04-18T21:52:19.828469+0000","last_undegraded":"2024-04-18T21:52:19.828469+0000","last_fullsized":"2024-04-18T21:52:19.828469+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:00.439865+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.21226381699999999,"stat_sum":{"num_bytes":4046848,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":2887,"num_write":18,"num_write_kb":4688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,10,2,5,11,8],"acting":[7,1,0,10,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.10","version":"59'26","reported_seq":146,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-18T21:52:15.865053+0000","last_change":"2024-04-18T21:52:15.867407+0000","last_active":"2024-04-18T21:52:15.865053+0000","last_peered":"2024-04-18T21:52:15.865053+0000","last_clean":"2024-04-18T21:52:15.865053+0000","last_became_active":"2024-04-18T21:52:01.384059+0000","last_became_peered":"2024-04-18T21:52:01.384059+0000","last_unstale":"2024-04-18T21:52:15.865053+0000","last_undegraded":"2024-04-18T21:52:15.865053+0000","last_fullsized":"2024-04-18T21:52:15.865053+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:18.295879+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0.30822618899999998,"stat_sum":{"num_bytes":3096576,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,11,2,4,10,7],"acting":[6,0,5,11,2,4,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"},{"start":"d","length":"1"}]},{"pgid":"3.11","version":"61'30","reported_seq":154,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.827816+0000","last_change":"2024-04-18T21:52:18.468562+0000","last_active":"2024-04-18T21:52:19.827816+0000","last_peered":"2024-04-18T21:52:19.827816+0000","last_clean":"2024-04-18T21:52:19.827816+0000","last_became_active":"2024-04-18T21:52:06.594871+0000","last_became_peered":"2024-04-18T21:52:06.594871+0000","last_unstale":"2024-04-18T21:52:19.827816+0000","last_undegraded":"2024-04-18T21:52:19.827816+0000","last_fullsized":"2024-04-18T21:52:19.827816+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'24","last_scrub_stamp":"2024-04-18T21:52:12.154768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:12.154768+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:38.775254+0000","scrub_duration":1591,"objects_trimmed":0,"snaptrim_duration":0.101665902,"stat_sum":{"num_bytes":3899392,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":5761,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,7],"acting":[3,4,6,10,1,5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.12","version":"56'11","reported_seq":109,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.727669+0000","last_change":"2024-04-18T21:52:18.463510+0000","last_active":"2024-04-18T21:52:19.727669+0000","last_peered":"2024-04-18T21:52:19.727669+0000","last_clean":"2024-04-18T21:52:19.727669+0000","last_became_active":"2024-04-18T21:52:11.882697+0000","last_became_peered":"2024-04-18T21:52:11.882697+0000","last_unstale":"2024-04-18T21:52:19.727669+0000","last_undegraded":"2024-04-18T21:52:19.727669+0000","last_fullsized":"2024-04-18T21:52:19.727669+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'11","last_scrub_stamp":"2024-04-18T21:52:15.060778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:15.060778+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:34.351219+0000","scrub_duration":799,"objects_trimmed":0,"snaptrim_duration":0.079362144999999995,"stat_sum":{"num_bytes":819200,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,10,0,5,6,1],"acting":[9,7,8,10,0,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]},{"pgid":"3.13","version":"62'47","reported_seq":150,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-18T21:52:19.728551+0000","last_change":"2024-04-18T21:52:18.463579+0000","last_active":"2024-04-18T21:52:19.728551+0000","last_peered":"2024-04-18T21:52:19.728551+0000","last_clean":"2024-04-18T21:52:19.728551+0000","last_became_active":"2024-04-18T21:52:16.145636+0000","last_became_peered":"2024-04-18T21:52:16.145636+0000","last_unstale":"2024-04-18T21:52:19.728551+0000","last_undegraded":"2024-04-18T21:52:19.728551+0000","last_fullsized":"2024-04-18T21:52:19.728551+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101046208,"stat_sum":{"num_bytes":3563520,"num_objects":10,"num_object_clones":4,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":25,"num_read_kb":3175,"num_write":22,"num_write_kb":4376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4,8,6,11,1,9],"acting":[7,5,4,8,6,11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"b","length":"4"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":53977088,"num_objects":73,"num_object_clones":23,"num_object_copies":584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":4,"num_read":133,"num_read_kb":27106,"num_write":170,"num_write_kb":33224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":105717760,"data_stored":118456320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":507,"ondisk_log_size":507,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378637,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38472,"kb_used_data":7288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333368,"statfs":{"total":96636764160,"available":96597368832,"internally_reserved":0,"allocated":7462912,"data_stored":8772460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378637,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41668,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330172,"statfs":{"total":96636764160,"available":96594096128,"internally_reserved":0,"allocated":10739712,"data_stored":12167787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378638,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":10188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":10432512,"data_stored":11512684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":12172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":12464128,"data_stored":13609836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378638,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39672,"kb_used_data":8488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332168,"statfs":{"total":96636764160,"available":96596140032,"internally_reserved":0,"allocated":8691712,"data_stored":9150007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411341,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33348,"kb_used_data":6324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":6475776,"data_stored":6663735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411341,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35484,"kb_used_data":8460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336356,"statfs":{"total":96636764160,"available":96600428544,"internally_reserved":0,"allocated":8663040,"data_stored":9760855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411341,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42436,"kb_used_data":11256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329404,"statfs":{"total":96636764160,"available":96593309696,"internally_reserved":0,"allocated":11526144,"data_stored":12136168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411342,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42848,"kb_used_data":11664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328992,"statfs":{"total":96636764160,"available":96592887808,"internally_reserved":0,"allocated":11943936,"data_stored":12746295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411340,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35624,"kb_used_data":8600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336216,"statfs":{"total":96636764160,"available":96600285184,"internally_reserved":0,"allocated":8806400,"data_stored":9367639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444045,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35688,"kb_used_data":8664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336152,"statfs":{"total":96636764160,"available":96600219648,"internally_reserved":0,"allocated":8871936,"data_stored":9498167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444045,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34876,"kb_used_data":7852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336964,"statfs":{"total":96636764160,"available":96601051136,"internally_reserved":0,"allocated":8040448,"data_stored":8805943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7766016,"data_stored":8622080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11800576,"data_stored":13225984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11608064,"data_stored":13029376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7991296,"data_stored":8740864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9768960,"data_stored":11128832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10813440,"data_stored":11718656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":10420224,"data_stored":11788288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7430144,"data_stored":8822784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":7372800,"data_stored":7868416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5775360,"data_stored":6254592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6799360,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8171520,"data_stored":8867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:20.791 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-04-18T21:52:20.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 16 2024-04-18T21:52:20.792 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:20.783+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:20.801 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 50 v 19 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-18T21:52:20.834 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:20.886 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:20.885+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s3 64/31 ECSubWrite(tid=47, reqid=osd.1.0:46, at_version=0'0, trim_to=0'0, roll_forward_to=64'28))", 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.871297+0000", 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.049859345999999999, 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049897410000000003, 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.920 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.871297+0000", 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.871297+0000", 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.871299+0000", 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1629999999999999e-06 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.871334+0000", 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4347999999999997e-05 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.871372+0000", 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7957000000000003e-05 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 64/31 ECSubWriteReply(tid=40, last_complete=64'8, committed=1, applied=1))", 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.881503+0000", 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.039653483000000003, 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039767729000000002, 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.923 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881503+0000", 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881503+0000", 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881504+0000", 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.313e-06 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881514+0000", 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6989999999999998e-06 2024-04-18T21:52:20.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881539+0000", 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5222e-05 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881556+0000", 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6466000000000001e-05 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.3s2 64/31 ECSubWrite(tid=154, reqid=client.4509.0:487, at_version=64'99, trim_to=0'0, roll_forward_to=62'92))", 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.881686+0000", 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.039470562000000001, 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039661816000000003, 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881686+0000", 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881686+0000", 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881687+0000", 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.37e-07 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881821+0000", 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013433400000000001 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881835+0000", 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4279e-05 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.881847+0000", 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1347e-05 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.929 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s5 64/58 ECSubWrite(tid=208, reqid=osd.7.0:206, at_version=0'0, trim_to=0'0, roll_forward_to=64'51))", 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.885430+0000", 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.035726295999999998, 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.035993085000000001, 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885430+0000", 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885430+0000", 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885432+0000", 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1969999999999999e-06 2024-04-18T21:52:20.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885450+0000", 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8512000000000002e-05 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885473+0000", 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2628000000000001e-05 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:20.932 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:20.933 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 4 2024-04-18T21:52:20.933 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:20.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:20.987 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:20.983+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:21.086 INFO:tasks.rados.rados.0.smithi097.stdout:197: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-18T21:52:21.088 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:21.083+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:21.092 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:21.092 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:21.092 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:21.092 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:21.093 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:21.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:21.136 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:21.137 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:21.170 INFO:tasks.rados.rados.0.smithi097.stdout:198: finishing write tid 1 to smithi09731216-19 2024-04-18T21:52:21.189 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:21.187+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:21.227 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:21.227 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:106 3.6s0 3:64491ca4:::smithi09731216-36:head [append 0~589824,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 34.137271745, 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.39237912, 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.228 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 106 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088750+0000", 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3859999999999999e-06 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089602+0000", 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000852537 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089629+0000", 2024-04-18T21:52:21.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6066000000000001e-05 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089648+0000", 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9449999999999998e-05 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174464+0000", 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816027000000002 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174469+0000", 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5970000000000002e-06 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481023+0000", 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3065548339999999 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481128+0000", 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010422299999999999 2024-04-18T21:52:21.232 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:107 3.6s0 3:64491ca4:::smithi09731216-36:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 34.136342782, 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.301432395, 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.233 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 107 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089679+0000", 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.686e-06 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089697+0000", 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7717e-05 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089714+0000", 2024-04-18T21:52:21.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7433000000000001e-05 2024-04-18T21:52:21.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089730+0000", 2024-04-18T21:52:21.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5953000000000001e-05 2024-04-18T21:52:21.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.237 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174547+0000", 2024-04-18T21:52:21.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816707000000005 2024-04-18T21:52:21.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174550+0000", 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9100000000000001e-06 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.983958+0000", 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8094078099999999 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984087+0000", 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012884800000000001 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984411+0000", 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032455500000000003 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986825+0000", 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024135609999999998 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391001+0000", 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40417611399999998 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391110+0000", 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109101 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.240 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:108 3.6s0 3:64491ca4:::smithi09731216-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 34.136269136999999, 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4199299170000002, 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 108 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089753+0000", 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.46e-06 2024-04-18T21:52:21.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089759+0000", 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5969999999999997e-06 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089782+0000", 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2546999999999999e-05 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089799+0000", 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6742999999999999e-05 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174653+0000", 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084854180000000001 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.886e-06 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391631+0000", 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2169758150000001 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391728+0000", 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6861000000000007e-05 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509681+0000", 2024-04-18T21:52:21.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.117952828 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:109 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~606208,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 34.136195583999999, 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4179864210000002, 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:21.246 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:21.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089826+0000", 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.147e-06 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090694+0000", 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00086783200000000002 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090726+0000", 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2379999999999998e-05 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090746+0000", 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9466999999999998e-05 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174599+0000", 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083853091000000005 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174600+0000", 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.097e-06 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985378+0000", 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.810778236 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985469+0000", 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0805999999999998e-05 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986312+0000", 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084315300000000002 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089266+0000", 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10295370099999999 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507611+0000", 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41834516199999999 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507811+0000", 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000200349 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:110 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:21.252 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 34.135252180999998, 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.429779607, 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 110 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090769+0000", 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090782+0000", 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2547e-05 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090797+0000", 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5109000000000001e-05 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090807+0000", 2024-04-18T21:52:21.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8670000000000006e-06 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083849218000000003 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174657+0000", 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8700000000000004e-07 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087432+0000", 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9127752569999998 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087519+0000", 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6668000000000004e-05 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088034+0000", 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00051538599999999995 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089552+0000", 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001518295 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520465+0000", 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.43091233000000001 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520548+0000", 2024-04-18T21:52:21.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2947000000000004e-05 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:111 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 34.135202442000001, 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5330603059999999, 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.259 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 111 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090819+0000", 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6000000000000003e-07 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090822+0000", 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.066e-06 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090834+0000", 2024-04-18T21:52:21.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2486999999999999e-05 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090844+0000", 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9899999999999992e-06 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275049+0000", 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.184205065 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275055+0000", 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4389999999999999e-06 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088362+0000", 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8133068830000001 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088621+0000", 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025974499999999997 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520968+0000", 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.432346972 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521091+0000", 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122263 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.623878+0000", 2024-04-18T21:52:21.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102787636 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:127 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~720896,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.411727456999998, 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50504046999999996, 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:21.265 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 127 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814297+0000", 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3120000000000004e-06 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815282+0000", 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0009845469999999999 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815306+0000", 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4488999999999999e-05 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815321+0000", 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4365999999999999e-05 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901001+0000", 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085680191000000003 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901126+0000", 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125506 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902346+0000", 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012201989999999999 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904676+0000", 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023299459999999998 2024-04-18T21:52:21.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319226+0000", 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.414549324 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319333+0000", 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010759 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:128 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.410672701000003, 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70473296200000002, 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 128 2024-04-18T21:52:21.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815349+0000", 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.003e-06 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815353+0000", 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5700000000000003e-06 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815362+0000", 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4540000000000007e-06 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815369+0000", 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4059999999999999e-06 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903371+0000", 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088001653999999999 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903511+0000", 2024-04-18T21:52:21.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000139889 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904181+0000", 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067014400000000001 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.905274+0000", 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010929430000000001 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519876+0000", 2024-04-18T21:52:21.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61460184500000004 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520081+0000", 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020505400000000001 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.1s0,GRANT,e:32,from:5(4),reservation_nonce:5)", 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.324486450999999, 2024-04-18T21:52:21.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61973115499999998, 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:21.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901535+0000", 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.206e-06 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901541+0000", 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.331e-06 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901559+0000", 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7977000000000001e-05 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.001690+0000", 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100130964 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521089+0000", 2024-04-18T21:52:21.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51939875099999999 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521142+0000", 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3324e-05 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521265+0000", 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122602 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:130 3.1s0 3:87c089d9:::smithi09731216-44:head [append 0~737280,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.315014181999999, 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61075046099999997, 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 130 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.281 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911008+0000", 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.356e-06 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911859+0000", 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085008399999999997 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911877+0000", 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8147000000000001e-05 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911886+0000", 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012129+0000", 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100243058 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012391+0000", 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026173100000000002 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.014631+0000", 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002240344 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118257+0000", 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103625859 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521564+0000", 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40330724200000001 2024-04-18T21:52:21.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521757+0000", 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019236000000000001 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:131 3.1s0 3:87c089d9:::smithi09731216-44:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.314115872999999, 2024-04-18T21:52:21.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70851564899999997, 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 131 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911905+0000", 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9800000000000002e-07 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911910+0000", 2024-04-18T21:52:21.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4499999999999997e-06 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911919+0000", 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911927+0000", 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5700000000000004e-06 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017049+0000", 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105122775 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017158+0000", 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010845600000000001 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017640+0000", 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00048220899999999999 2024-04-18T21:52:21.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118261+0000", 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100621063 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620272+0000", 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50201088400000005 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620420+0000", 2024-04-18T21:52:21.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000148064 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 32/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.206251217999998, 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29966542299999999, 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019769+0000", 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019772+0000", 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2869999999999999e-06 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019780+0000", 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8990000000000001e-06 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.019783+0000", 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8460000000000001e-06 2024-04-18T21:52:21.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319409+0000", 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29962544499999999 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319435+0000", 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5605999999999998e-05 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.295 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.203992425999999, 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.297022972, 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022028+0000", 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022030+0000", 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7940000000000001e-06 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022039+0000", 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6339999999999996e-06 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022063+0000", 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4026e-05 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.022074+0000", 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0705e-05 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319012+0000", 2024-04-18T21:52:21.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29693840300000002 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319051+0000", 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.841e-05 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 33/31 ECSubWriteReply(tid=15, last_complete=32'3, committed=1, applied=1))", 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.136563809999998, 2024-04-18T21:52:21.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30109323399999999, 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089457+0000", 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089458+0000", 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5859999999999999e-06 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089467+0000", 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0420000000000008e-06 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089492+0000", 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5211999999999998e-05 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.189619+0000", 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100126885 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.390510+0000", 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20089099299999999 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.390550+0000", 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9515999999999999e-05 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.119131431, 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31260053300000001, 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106890+0000", 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8599999999999996e-07 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106896+0000", 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9139999999999996e-06 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106911+0000", 2024-04-18T21:52:21.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4800999999999999e-05 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106920+0000", 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9379999999999998e-06 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419470+0000", 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.312550204 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419489+0000", 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.969e-05 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.021206657999997, 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.316195273, 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.307 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1699999999999998e-07 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204818+0000", 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5659999999999998e-06 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204828+0000", 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0478e-05 2024-04-18T21:52:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204836+0000", 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.413e-06 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519654+0000", 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.314818722 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521009+0000", 2024-04-18T21:52:21.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001354577 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=13, last_complete=32'5, committed=1, applied=1))", 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.021172972999999, 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31487986099999998, 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.311 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204848+0000", 2024-04-18T21:52:21.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8299999999999998e-07 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204850+0000", 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6429999999999999e-06 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204859+0000", 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0550000000000005e-06 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204866+0000", 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5320000000000003e-06 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519669+0000", 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31480259199999999 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519727+0000", 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8656000000000002e-05 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.314 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.021140115999998, 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31508690700000003, 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204881+0000", 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3000000000000002e-07 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204884+0000", 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.976e-06 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204900+0000", 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6418000000000001e-05 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204908+0000", 2024-04-18T21:52:21.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3080000000000008e-06 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519923+0000", 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31501511199999999 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519967+0000", 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3763000000000001e-05 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.018390939, 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31221361399999997, 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:21.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207629+0000", 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207631+0000", 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.42e-06 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.320 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207640+0000", 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7350000000000002e-06 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207668+0000", 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8881999999999999e-05 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.207686+0000", 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7136999999999999e-05 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519798+0000", 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31211196099999999 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519843+0000", 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5479000000000003e-05 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.322 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es5 64/26 ECSubWrite(tid=48, reqid=osd.2.0:48, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.579562+0000", 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.64645814199999996, 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.295164496, 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.579562+0000", 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.579562+0000", 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.579562+0000", 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5100000000000001e-07 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.579583+0000", 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1021000000000001e-05 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.579606+0000", 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2736000000000001e-05 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.679688+0000", 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10008208 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.679804+0000", 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115812 2024-04-18T21:52:21.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.679820+0000", 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5942000000000001e-05 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.874658+0000", 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19483811200000001 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:21.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.874727+0000", 2024-04-18T21:52:21.327 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8541999999999995e-05 2024-04-18T21:52:21.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.327 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:21.327 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:21.327 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:21.329 INFO:tasks.rados.rados.0.smithi097.stdout:198: finishing write tid 2 to smithi09731216-19 2024-04-18T21:52:21.329 INFO:tasks.rados.rados.0.smithi097.stdout:198: finishing write tid 3 to smithi09731216-19 2024-04-18T21:52:21.329 INFO:tasks.rados.rados.0.smithi097.stdout:194: finishing write tid 1 to smithi09731216-6 2024-04-18T21:52:21.329 INFO:tasks.rados.rados.0.smithi097.stdout:194: finishing write tid 2 to smithi09731216-6 2024-04-18T21:52:21.329 INFO:tasks.rados.rados.0.smithi097.stdout:194: finishing write tid 3 to smithi09731216-6 2024-04-18T21:52:21.329 INFO:tasks.rados.rados.0.smithi097.stdout:194: finishing write tid 4 to smithi09731216-6 2024-04-18T21:52:21.330 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:21.329+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:21.330 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:21.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:21.331 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:21.362 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:21.362 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:21.372 INFO:tasks.rados.rados.0.smithi097.stdout:198: finishing write tid 4 to smithi09731216-19 2024-04-18T21:52:21.375 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:21.371+0000 7f3d90efa640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 9 after trimming was started 2024-04-18T21:52:21.401 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:21.402 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:21.418 INFO:tasks.rados.rados.0.smithi097.stdout:195: finishing write tid 1 to smithi09731216-20 2024-04-18T21:52:21.419 INFO:tasks.rados.rados.0.smithi097.stdout:195: finishing write tid 2 to smithi09731216-20 2024-04-18T21:52:21.430 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:21.427+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:21.437 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:21.437 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:21.460 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:21.460 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:21.471 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:21.471 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:21.472 INFO:teuthology.orchestra.run.smithi097.stderr:set pool 3 pg_num to 16 2024-04-18T21:52:21.474 INFO:tasks.rados.rados.0.smithi097.stdout:196: done (5 left) 2024-04-18T21:52:21.475 INFO:tasks.rados.rados.0.smithi097.stdout:197: done (4 left) 2024-04-18T21:52:21.475 INFO:tasks.rados.rados.0.smithi097.stdout:199: done (3 left) 2024-04-18T21:52:21.475 INFO:tasks.rados.rados.0.smithi097.stdout:200: snap_create 2024-04-18T21:52:21.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:21.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:52:21.499 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:21.500 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:21.530 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:21.528+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:21.576 INFO:tasks.rados.rados.0.smithi097.stdout:198: finishing write tid 5 to smithi09731216-19 2024-04-18T21:52:21.576 INFO:tasks.rados.rados.0.smithi097.stdout:198: oid 19 updating version 0 to 33 2024-04-18T21:52:21.576 INFO:tasks.rados.rados.0.smithi097.stdout:198: oid 19 updating version 33 to 34 2024-04-18T21:52:21.576 INFO:tasks.rados.rados.0.smithi097.stdout:198: oid 19 updating version 34 to 35 2024-04-18T21:52:21.576 INFO:tasks.rados.rados.0.smithi097.stdout:198: oid 19 version 35 is already newer than 32 2024-04-18T21:52:21.576 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 19 v 35 (ObjNum 89 snap 15 seq_num 89) dirty exists 2024-04-18T21:52:21.576 INFO:tasks.rados.rados.0.smithi097.stdout:198: left oid 19 (ObjNum 89 snap 15 seq_num 89) 2024-04-18T21:52:21.615 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:21.615 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:21.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:21.631 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:21.627+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:21.726 INFO:tasks.rados.rados.0.smithi097.stdout:194: finishing write tid 5 to smithi09731216-6 2024-04-18T21:52:21.726 INFO:tasks.rados.rados.0.smithi097.stdout:194: oid 6 updating version 0 to 93 2024-04-18T21:52:21.726 INFO:tasks.rados.rados.0.smithi097.stdout:194: oid 6 updating version 93 to 96 2024-04-18T21:52:21.726 INFO:tasks.rados.rados.0.smithi097.stdout:194: oid 6 version 96 is already newer than 94 2024-04-18T21:52:21.726 INFO:tasks.rados.rados.0.smithi097.stdout:194: oid 6 version 96 is already newer than 95 2024-04-18T21:52:21.726 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 6 v 96 (ObjNum 87 snap 15 seq_num 87) dirty exists 2024-04-18T21:52:21.726 INFO:tasks.rados.rados.0.smithi097.stdout:194: left oid 6 (ObjNum 87 snap 15 seq_num 87) 2024-04-18T21:52:21.731 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:21.728+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:21.790 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:21.790 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:21.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:21.832 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:21.832+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:21.920 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:21.920 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:21.934 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:21.932+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:21.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:21.984 INFO:tasks.rados.rados.0.smithi097.stdout:195: finishing write tid 3 to smithi09731216-20 2024-04-18T21:52:22.035 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:22.032+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:22.057 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:22.057 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:22.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:22.085 INFO:tasks.rados.rados.0.smithi097.stdout:195: finishing write tid 4 to smithi09731216-20 2024-04-18T21:52:22.135 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:22.131+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:22.195 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:22.195 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:22.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:22.237 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:22.236+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:22.337 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:22.335+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:22.356 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:52:22.361 INFO:tasks.rados.rados.0.smithi097.stdout:194: done (3 left) 2024-04-18T21:52:22.361 INFO:tasks.rados.rados.0.smithi097.stdout:198: done (2 left) 2024-04-18T21:52:22.361 INFO:tasks.rados.rados.0.smithi097.stdout:200: done (1 left) 2024-04-18T21:52:22.361 INFO:tasks.rados.rados.0.smithi097.stdout:201: read oid 6 snap -1 2024-04-18T21:52:22.362 INFO:tasks.rados.rados.0.smithi097.stdout:201: expect (ObjNum 87 snap 15 seq_num 87) 2024-04-18T21:52:22.362 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 12 current snap is 17 2024-04-18T21:52:22.362 INFO:tasks.rados.rados.0.smithi097.stdout:202: seq_num 90 ranges {1409024=753664,2162688=720896,2883584=49152} 2024-04-18T21:52:22.378 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:22.378 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:22.379 INFO:tasks.rados.rados.0.smithi097.stdout:202: writing smithi09731216-12 from 1409024 to 2162688 tid 1 2024-04-18T21:52:22.384 INFO:tasks.rados.rados.0.smithi097.stdout:202: writing smithi09731216-12 from 2162688 to 2883584 tid 2 2024-04-18T21:52:22.385 INFO:tasks.rados.rados.0.smithi097.stdout:202: writing smithi09731216-12 from 2883584 to 2932736 tid 3 2024-04-18T21:52:22.385 INFO:tasks.rados.rados.0.smithi097.stdout:203: read oid 32 snap -1 2024-04-18T21:52:22.385 INFO:tasks.rados.rados.0.smithi097.stdout:203: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:52:22.385 INFO:tasks.rados.rados.0.smithi097.stdout:204: read oid 9 snap -1 2024-04-18T21:52:22.385 INFO:tasks.rados.rados.0.smithi097.stdout:204: expect (ObjNum 71 snap 5 seq_num 71) 2024-04-18T21:52:22.385 INFO:tasks.rados.rados.0.smithi097.stdout:205: read oid 24 snap -1 2024-04-18T21:52:22.385 INFO:tasks.rados.rados.0.smithi097.stdout:205: expect (ObjNum 59 snap 3 seq_num 59) 2024-04-18T21:52:22.385 INFO:tasks.rados.rados.0.smithi097.stdout:206: snap_create 2024-04-18T21:52:22.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:22.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:22.435+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:22.464 INFO:tasks.rados.rados.0.smithi097.stdout:195: finishing write tid 5 to smithi09731216-20 2024-04-18T21:52:22.464 INFO:tasks.rados.rados.0.smithi097.stdout:195: oid 20 updating version 0 to 99 2024-04-18T21:52:22.464 INFO:tasks.rados.rados.0.smithi097.stdout:195: oid 20 updating version 99 to 101 2024-04-18T21:52:22.465 INFO:tasks.rados.rados.0.smithi097.stdout:195: oid 20 version 101 is already newer than 100 2024-04-18T21:52:22.465 INFO:tasks.rados.rados.0.smithi097.stdout:195: oid 20 version 101 is already newer than 98 2024-04-18T21:52:22.465 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 20 v 101 (ObjNum 88 snap 15 seq_num 88) dirty exists 2024-04-18T21:52:22.465 INFO:tasks.rados.rados.0.smithi097.stdout:195: left oid 20 (ObjNum 88 snap 15 seq_num 88) 2024-04-18T21:52:22.466 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.459+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.466 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.466 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.466 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.466 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.466 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.466 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.468 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.469 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.469 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.469 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.470 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.470 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.470 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.471 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.474 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.474 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.474 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.474 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.474 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.474 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.474 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.474 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.475 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.463+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.476 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.477 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.467+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.539 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:22.535+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:22.563 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:22.564 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:22.582 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.584 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.567+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.585 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.586 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.587 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.588 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.590 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.591 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.571+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.592 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.594 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.594 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.594 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.575+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:22.639 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:22.635+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:22.680 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.680 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.681 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.682 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.683 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.683 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.683 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.683 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:22.675+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:22.703 INFO:tasks.rados.rados.0.smithi097.stdout:201: expect (ObjNum 87 snap 15 seq_num 87) 2024-04-18T21:52:22.740 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:22.735+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:22.763 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:22.763 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:22.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:22.812 INFO:tasks.rados.rados.0.smithi097.stdout:204: expect (ObjNum 71 snap 5 seq_num 71) 2024-04-18T21:52:22.857 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:22.855+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:22.914 INFO:tasks.rados.rados.0.smithi097.stdout:203: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:52:22.934 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:22.934 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:22.942 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:22.939+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:22.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:22.982 INFO:tasks.rados.rados.0.smithi097.stdout:202: finishing write tid 1 to smithi09731216-12 2024-04-18T21:52:22.982 INFO:tasks.rados.rados.0.smithi097.stdout:202: finishing write tid 2 to smithi09731216-12 2024-04-18T21:52:22.984 INFO:tasks.rados.rados.0.smithi097.stdout:202: finishing write tid 3 to smithi09731216-12 2024-04-18T21:52:23.042 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:23.039+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:23.107 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:23.107 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:23.112 INFO:tasks.rados.rados.0.smithi097.stdout:205: expect (ObjNum 59 snap 3 seq_num 59) 2024-04-18T21:52:23.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:23.169 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:23.167+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:23.246 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:23.243+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:23.295 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:23.295 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:23.310 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:23.347 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:23.344+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:23.362 INFO:tasks.rados.rados.0.smithi097.stdout:195: done (6 left) 2024-04-18T21:52:23.362 INFO:tasks.rados.rados.0.smithi097.stdout:201: done (5 left) 2024-04-18T21:52:23.363 INFO:tasks.rados.rados.0.smithi097.stdout:203: done (4 left) 2024-04-18T21:52:23.363 INFO:tasks.rados.rados.0.smithi097.stdout:204: done (3 left) 2024-04-18T21:52:23.363 INFO:tasks.rados.rados.0.smithi097.stdout:205: done (2 left) 2024-04-18T21:52:23.363 INFO:tasks.rados.rados.0.smithi097.stdout:206: done (1 left) 2024-04-18T21:52:23.363 INFO:tasks.rados.rados.0.smithi097.stdout:207: snap_remove snap 13 2024-04-18T21:52:23.447 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:23.443+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:23.448 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:23.448 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:23.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:23.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:23.548 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:23.548+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:23.582 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:23.582 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:23.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:23.649 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:23.648+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:23.674 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:23.674 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:23.690 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:23.706 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:23.707 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:23.714 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:23.714 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:23.725 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:23.725 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:23.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:23.744 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:23.744 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:23.750 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:23.747+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:23.761 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:23.761 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:23.779 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:23.779 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:23.796 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:23.796 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:23.817 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:23.817 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:23.837 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:23.838 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:23.851 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:23.848+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPush(3.14s7 67/65 [PushOp(3:2c76b789:::smithi09731216-42:head, version: 51'8, data_included: [0~16384], data_size: 16384, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:2c76b789:::smithi09731216-42:head@51'8, size: 65536, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:23.772411+0000", 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.082870256000000003, 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082968895000000001, 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:23.855 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:23.772411+0000", 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:23.772411+0000", 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:23.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:23.772412+0000", 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0210000000000001e-06 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:23.772450+0000", 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7970999999999999e-05 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:23.772476+0000", 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6064999999999999e-05 2024-04-18T21:52:23.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:23.772498+0000", 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1761999999999999e-05 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 1 2024-04-18T21:52:23.858 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:23.859 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:23.859 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:23.870 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:23.872 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:23.867+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:23.881 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:23.881 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:23.952 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:23.948+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:23.991 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:23.991 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:23.991 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:23.991 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:23.991 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:24.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:24.053 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:24.051+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:100 3.cs0 3:348e44d8:::smithi09731216-34:head [append 0~557056,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.463026089000003, 2024-04-18T21:52:24.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50652407300000002, 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:24.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659953+0000", 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.821e-06 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660381+0000", 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042725800000000002 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660395+0000", 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.394e-05 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660404+0000", 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6520000000000001e-06 2024-04-18T21:52:24.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760564+0000", 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10015995900000001 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760699+0000", 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013526 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762105+0000", 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00140595 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965039+0000", 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20293309400000001 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166421+0000", 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201382373 2024-04-18T21:52:24.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166476+0000", 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4765999999999997e-05 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:101 3.cs0 3:348e44d8:::smithi09731216-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.462556349000003, 2024-04-18T21:52:24.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60956795200000002, 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660422+0000", 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-07 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660426+0000", 2024-04-18T21:52:24.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3830000000000002e-06 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660436+0000", 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8679999999999994e-06 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660443+0000", 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2999999999999998e-06 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762872+0000", 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1024294 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762973+0000", 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001015 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763506+0000", 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053209199999999996 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965047+0000", 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20154100999999999 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269839+0000", 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.304792377 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269989+0000", 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015033399999999999 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:102 3.cs0 3:348e44d8:::smithi09731216-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.4625068, 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.92759091000000005, 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.138 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660472+0000", 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660476+0000", 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0879999999999997e-06 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660493+0000", 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6963e-05 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660502+0000", 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9579999999999996e-06 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763793+0000", 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10329121600000001 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763891+0000", 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8127000000000004e-05 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372039+0000", 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60814838199999999 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372189+0000", 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014991200000000001 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.588062+0000", 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.215872654 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.5s0 30/26 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.363112311999998, 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30082227499999997, 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:52:24.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759866+0000", 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.34e-07 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759873+0000", 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.973e-06 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759879+0000", 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5440000000000004e-06 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759885+0000", 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3340000000000001e-06 2024-04-18T21:52:24.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.060439+0000", 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30055447000000002 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.060688+0000", 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024862000000000002 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.357345770999999, 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299971873, 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1800000000000001e-07 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765635+0000", 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6450000000000001e-06 2024-04-18T21:52:24.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765644+0000", 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.755e-06 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.865734+0000", 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10008987699999999 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065540+0000", 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19980606200000001 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065604+0000", 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4115999999999994e-05 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.256283015999998, 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299440857, 2024-04-18T21:52:24.151 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.13e-07 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866703+0000", 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866722+0000", 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9522999999999999e-05 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966809+0000", 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10008695400000001 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166119+0000", 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19931021099999999 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166136+0000", 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6319000000000001e-05 2024-04-18T21:52:24.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:78 3.15s0 3:ab33d1b5:::smithi09731216-26:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.141961375999998, 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1076682130000002, 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981019+0000", 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.447e-06 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981025+0000", 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6810000000000001e-06 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981044+0000", 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8856999999999999e-05 2024-04-18T21:52:24.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981056+0000", 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1921e-05 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164029+0000", 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18297308600000001 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:24.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164033+0000", 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0369999999999996e-06 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885490+0000", 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7214569900000001 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885771+0000", 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028064300000000002 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.088685+0000", 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.202913551 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:133 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.112695379000002, 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.393294378, 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 133 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010286+0000", 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012871+0000", 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025854699999999999 2024-04-18T21:52:24.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012896+0000", 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5409e-05 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012911+0000", 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.484e-05 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013086+0000", 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017422499999999999 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013437+0000", 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035166100000000002 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.016765+0000", 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033273949999999999 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.020122+0000", 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033576370000000001 2024-04-18T21:52:24.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403436+0000", 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38331330800000002 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403577+0000", 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014114999999999999 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:134 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.109993852999999, 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39119692299999997, 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 134 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.166 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012986+0000", 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0839999999999999e-06 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012995+0000", 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4159999999999993e-06 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013025+0000", 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0035999999999998e-05 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013046+0000", 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0720000000000002e-05 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.017993+0000", 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049471020000000001 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.018258+0000", 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026496000000000001 2024-04-18T21:52:24.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.019560+0000", 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001301373 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.020585+0000", 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010257910000000001 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403896+0000", 2024-04-18T21:52:24.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38331068899999998 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.404181+0000", 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028475199999999999 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:237 3.cs0 3:3bc99ad3:::smithi09731216-27:head [read 0~442368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.089733283000001, 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30619164900000001, 2024-04-18T21:52:24.171 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033245+0000", 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3499999999999996e-07 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033248+0000", 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3730000000000001e-06 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033256+0000", 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8509999999999999e-06 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033262+0000", 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4480000000000002e-06 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033435+0000", 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000173847 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033439+0000", 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6540000000000001e-06 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.237425+0000", 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20398589 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.237506+0000", 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1211000000000006e-05 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.339436+0000", 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10192983999999999 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 40/26 ECSubWriteReply(tid=53, last_complete=40'21, committed=1, applied=1))", 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.601352816999999, 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.409140051, 2024-04-18T21:52:24.176 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0900000000000002e-07 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521629+0000", 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0169999999999998e-06 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521640+0000", 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0246e-05 2024-04-18T21:52:24.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521647+0000", 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9070000000000001e-06 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.929171+0000", 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40752403300000001 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.930765+0000", 2024-04-18T21:52:24.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001594339 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:463 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~638976] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7488885270000001, 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50349991299999997, 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.180 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 463 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374091+0000", 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.84e-06 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375137+0000", 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010459 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375156+0000", 2024-04-18T21:52:24.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978e-05 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375169+0000", 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2660000000000001e-05 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571128+0000", 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19595929000000001 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571311+0000", 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018294599999999999 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572699+0000", 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001388715 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675602+0000", 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102902721 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877476+0000", 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20187386299999999 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877589+0000", 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:464 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~131072] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7477749420000004, 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50427034999999998, 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 464 2024-04-18T21:52:24.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375204+0000", 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3650000000000001e-06 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375344+0000", 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013967800000000001 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375357+0000", 2024-04-18T21:52:24.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3580999999999999e-05 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375367+0000", 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0117e-05 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573497+0000", 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198129889 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573606+0000", 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010875 2024-04-18T21:52:24.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574480+0000", 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00087374900000000005 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675605+0000", 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10112497400000001 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879333+0000", 2024-04-18T21:52:24.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20372839500000001 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879473+0000", 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013985199999999999 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:465 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:24.191 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.747592407, 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60269858399999998, 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 465 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:24.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375386+0000", 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6300000000000004e-07 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375391+0000", 2024-04-18T21:52:24.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.729e-06 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375401+0000", 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9569999999999997e-06 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375409+0000", 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6620000000000005e-06 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777143+0000", 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40173476200000002 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777239+0000", 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5712999999999997e-05 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777695+0000", 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045595 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.778376+0000", 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068123800000000003 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.977952+0000", 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.199575432 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.978084+0000", 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000132278 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.196 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:466 3.cs0 3:3c73a4f8:::smithi09731216-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7475577199999996, 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006726617, 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 466 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.197 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375421+0000", 2024-04-18T21:52:24.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.009e-06 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375424+0000", 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.495e-06 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375435+0000", 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0360000000000001e-05 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375442+0000", 2024-04-18T21:52:24.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6389999999999992e-06 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777929+0000", 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40248657999999998 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777993+0000", 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078687+0000", 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30069407399999998 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078849+0000", 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162065 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.382147+0000", 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30329795300000001 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.201 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 61/26 ECSubWriteReply(tid=88, last_complete=62'48, committed=1, applied=1))", 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.379271+0000", 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7437067939999995, 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29601272699999998, 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379271+0000", 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379271+0000", 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379272+0000", 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0559999999999999e-06 2024-04-18T21:52:24.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379277+0000", 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7010000000000004e-06 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379289+0000", 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2799999999999999e-05 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379297+0000", 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5039999999999996e-06 2024-04-18T21:52:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675220+0000", 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29592301700000001 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675284+0000", 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3649e-05 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.205 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 62/26 ECSubWriteReply(tid=88, last_complete=62'48, committed=1, applied=1))", 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.380752+0000", 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7422254859999997, 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29469121700000001, 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380752+0000", 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380752+0000", 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380753+0000", 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3600000000000003e-07 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380757+0000", 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.126e-06 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380769+0000", 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2135e-05 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380778+0000", 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7190000000000007e-06 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675386+0000", 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29460853199999998 2024-04-18T21:52:24.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675443+0000", 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6969000000000003e-05 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 62/26 ECSubWriteReply(tid=88, last_complete=62'48, committed=1, applied=1))", 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.384200+0000", 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7387776060000002, 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29131686099999998, 2024-04-18T21:52:24.209 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.384200+0000", 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.384200+0000", 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.384201+0000", 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4499999999999997e-07 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.384204+0000", 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.602e-06 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.384215+0000", 2024-04-18T21:52:24.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0823e-05 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.384224+0000", 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6070000000000008e-06 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675478+0000", 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29125469399999998 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675517+0000", 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8489999999999999e-05 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.11s5 62/49 ECSubRead(tid=94, to_read={3:8fad7925:::smithi09731216-43:head=0,491520,0}, subchunks={3:8fad7925:::smithi09731216-43:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.571167+0000", 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.5518105220000002, 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.300446565, 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.213 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571167+0000", 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571167+0000", 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571168+0000", 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1800000000000005e-07 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571178+0000", 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0111e-05 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571190+0000", 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2016e-05 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.671270+0000", 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10007946099999999 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.871478+0000", 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200207947 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.871614+0000", 2024-04-18T21:52:24.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000136212 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:491 3.5s0 3:a2c4e662:::smithi09731216-50:head [create,setxattr `k]abUjc_kPO]\\rQ[m\\eRa\\f]aW]pgQYXV]gVkgc (1435) in=40b,setxattr qen`h]Vg]rpmXc_]tPY^Xpfd (1823) in=24b,setxattr [idOh[aNZjbdcYS\\eZaVVetqi[ric_V (1823) in=31b,setxattr daSce\\mdeZpfaaqjVU^^]PU\\S`[RbRZmXqSkgZQj (1087) in=40b,setxattr YgloTThONOYj[cntZofPOaP^O_bsWg]fd (1823) in=33b,setxattr ejbfqsafT_htneSabdatV`Qp^RS`f[nN``fN\\ebj (807) in=40b,setxattr Ya\\[Sha^pYao_k[dboRNsfOYcXoQtQRfV[XWcTQj (947) in=40b,setxattr YNVc]ZTPiOoc`NXSYsd[NZljlaqNVlYlRnihjYdQ (1789) in=40b,setxattr _t]sstiO]naqp_Zar\\Vtqss]cmN]kYnT_NtflgVd (9120) in=40b,setxattr lXVOWmpdrdYlUkq]]nPfdVVotggPsOZ\\VmPemila (909) in=40b,setxattr slQjq^hbmpO^nVhe\\te\\SkYldVebmeasZVgapUqn (454) in=40b,setxattr n[ea[_dQdc][]\\bfthkjd\\aSbNS^Qe_ZX\\Wr_^^V (594) in=40b,setxattr nS]tbdRhgf]l`\\Tfrnkihlk`Zpgbn[PY`anPZnhr (1682) in=40b,setxattr rpSrTfUUP^cVp_\\d]tNQnmXf[Np\\eO_YeldoslW_ (946) in=40b,setxattr VePmePiYj[hSVdatcprSk\\m]]tXhjbke]jZZYRQS (1681) in=40b,setxattr or[\\eoSZ^ZtXlcRqVTioQ[kZUnqnW[\\ (1823) in=31b,setxattr SS\\`X_`sgfdYjRdpNccP`gcPcRp^eSYXXOOs\\dN[ (1821) in=40b,setxattr obshrdURd^[OaVZZiSTRdPe^cqqgmSO[^Spi^Sgh (1821) in=40b,setxattr oUcnoPgXaRjdRatSbsPd[Ts]f\\pgbOl_Qk^R^Nfq (12280) in=40b,setxattr Xsgptojf]Obssrjj[_NfoopYpXifZhOd\\_]gWoXe (5052) in=40b] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.495516+0000", 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.6274615749999999, 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30682174400000001, 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.217 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 491 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495516+0000", 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495516+0000", 2024-04-18T21:52:24.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495520+0000", 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6600000000000001e-06 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495676+0000", 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015610099999999999 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495713+0000", 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7406000000000001e-05 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495736+0000", 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.249e-05 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495905+0000", 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000169204 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.496234+0000", 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032931200000000001 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.497997+0000", 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001762493 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.500587+0000", 2024-04-18T21:52:24.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025899999999999999 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.802032+0000", 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30144495900000001 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.802338+0000", 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030611899999999999 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.222 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.223 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.223 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:24.224 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:24.224 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:24.219+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:24.325 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:24.323+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:24.367 INFO:tasks.rados.rados.0.smithi097.stdout:207: done (1 left) 2024-04-18T21:52:24.367 INFO:tasks.rados.rados.0.smithi097.stdout:208: read oid 6 snap -1 2024-04-18T21:52:24.367 INFO:tasks.rados.rados.0.smithi097.stdout:208: expect (ObjNum 87 snap 15 seq_num 87) 2024-04-18T21:52:24.367 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 37 current snap is 18 2024-04-18T21:52:24.367 INFO:tasks.rados.rados.0.smithi097.stdout:209: seq_num 91 ranges {606208=770048,1376256=475136,1851392=294912} 2024-04-18T21:52:24.378 INFO:tasks.rados.rados.0.smithi097.stdout:209: writing smithi09731216-37 from 606208 to 1376256 tid 1 2024-04-18T21:52:24.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-18T21:52:24.381 INFO:tasks.rados.rados.0.smithi097.stdout:209: writing smithi09731216-37 from 1376256 to 1851392 tid 2 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:209: writing smithi09731216-37 from 1851392 to 2146304 tid 3 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 20 current snap is 18 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:202: finishing write tid 4 to smithi09731216-12 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:202: finishing write tid 5 to smithi09731216-12 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:202: oid 12 updating version 0 to 37 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:202: oid 12 updating version 37 to 40 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:202: oid 12 version 40 is already newer than 39 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:202: oid 12 version 40 is already newer than 38 2024-04-18T21:52:24.384 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 12 v 40 (ObjNum 90 snap 17 seq_num 90) dirty exists 2024-04-18T21:52:24.385 INFO:tasks.rados.rados.0.smithi097.stdout:202: left oid 12 (ObjNum 90 snap 17 seq_num 90) 2024-04-18T21:52:24.385 INFO:tasks.rados.rados.0.smithi097.stdout:210: seq_num 92 ranges {2818048=786432,3604480=491520,4096000=278528} 2024-04-18T21:52:24.404 INFO:tasks.rados.rados.0.smithi097.stdout:210: writing smithi09731216-20 from 2818048 to 3604480 tid 1 2024-04-18T21:52:24.406 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:24.406 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:24.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.406 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s3 68/31 ECSubWrite(tid=170, reqid=osd.4.0:170, at_version=68'107, trim_to=0'0, roll_forward_to=65'105))", 2024-04-18T21:52:24.406 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.374599+0000", 2024-04-18T21:52:24.406 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.031237199, 2024-04-18T21:52:24.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031293556, 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.374599+0000", 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.374599+0000", 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.374599+0000", 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8699999999999995e-07 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.374686+0000", 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6738999999999997e-05 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.374721+0000", 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4798999999999997e-05 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.374746+0000", 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4896999999999999e-05 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.374921+0000", 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017524399999999999 2024-04-18T21:52:24.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.374953+0000", 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.222e-05 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.16s5 68/31 ECSubWrite(tid=42, reqid=client.4509.0:524, at_version=68'5, trim_to=0'0, roll_forward_to=32'3))", 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.389675+0000", 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.016160643999999998, 2024-04-18T21:52:24.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016409975, 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.389675+0000", 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.389675+0000", 2024-04-18T21:52:24.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.389676+0000", 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0100000000000001e-06 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.390245+0000", 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056902999999999999 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.390299+0000", 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3742000000000001e-05 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.390329+0000", 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9697e-05 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.16s5 68/31 ECSubWrite(tid=43, reqid=client.4509.0:525, at_version=68'6, trim_to=0'0, roll_forward_to=32'3))", 2024-04-18T21:52:24.413 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.396518+0000", 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0093174639999999993, 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.009714706, 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.396518+0000", 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.396518+0000", 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.396520+0000", 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5230000000000001e-06 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.396822+0000", 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030252600000000001 2024-04-18T21:52:24.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.396857+0000", 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4356000000000001e-05 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.396884+0000", 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7529999999999999e-05 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.416 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.16s5 68/31 ECSubWrite(tid=44, reqid=client.4509.0:526, at_version=68'7, trim_to=0'0, roll_forward_to=32'3))", 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.399398+0000", 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0064380679999999999, 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006976828, 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.399398+0000", 2024-04-18T21:52:24.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.399398+0000", 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.399399+0000", 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.124e-06 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.399578+0000", 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017907099999999999 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.399598+0000", 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9956999999999999e-05 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.399610+0000", 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1836e-05 2024-04-18T21:52:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.16s5 68/31 ECSubWrite(tid=45, reqid=client.4509.0:527, at_version=68'8, trim_to=0'0, roll_forward_to=32'3))", 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.401331+0000", 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0045049749999999996, 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051351770000000003, 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.401331+0000", 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.401331+0000", 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.401332+0000", 2024-04-18T21:52:24.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3799999999999999e-06 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.401364+0000", 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1742999999999997e-05 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.401382+0000", 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8338999999999998e-05 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.401393+0000", 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1063999999999999e-05 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 68/31 ECSubWrite(tid=46, reqid=osd.11.0:46, at_version=68'10, trim_to=0'0, roll_forward_to=64'8))", 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.404684+0000", 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0011519709999999999, 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018544259999999999, 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.423 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.404684+0000", 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.404684+0000", 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.404686+0000", 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8360000000000001e-06 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.404726+0000", 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0606999999999999e-05 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.404753+0000", 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7049000000000001e-05 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.404771+0000", 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7963000000000001e-05 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 6 2024-04-18T21:52:24.426 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:24.428 INFO:tasks.rados.rados.0.smithi097.stdout:210: writing smithi09731216-20 from 3604480 to 4096000 tid 2 2024-04-18T21:52:24.428 INFO:tasks.rados.rados.0.smithi097.stdout:210: writing smithi09731216-20 from 4096000 to 4374528 tid 3 2024-04-18T21:52:24.428 INFO:tasks.rados.rados.0.smithi097.stdout:202: done (3 left) 2024-04-18T21:52:24.428 INFO:tasks.rados.rados.0.smithi097.stdout:211: rmattr oid 10 current snap is 18 2024-04-18T21:52:24.428 INFO:tasks.rados.rados.0.smithi097.stdout:211: done (3 left) 2024-04-18T21:52:24.428 INFO:tasks.rados.rados.0.smithi097.stdout:212: snap_remove snap 17 2024-04-18T21:52:24.429 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:24.429 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:24.423+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:24.528 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:24.527+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:24.599 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:24.600 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:24.600 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:24.600 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:24.600 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:24.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:24.657 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:24.655+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:24.730 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:24.727+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:24.779 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:24.779 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:24.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs7 36/26 ECSubWrite(tid=36, reqid=client.4509.0:216, at_version=36'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.757953454999999, 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30205913200000001, 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018545+0000", 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.35e-07 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018570+0000", 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.438e-05 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.018583+0000", 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2928999999999999e-05 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.118702+0000", 2024-04-18T21:52:24.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1001194 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218952+0000", 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10025022 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.218966+0000", 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3709e-05 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320525+0000", 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10155911099999999 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.320604+0000", 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9147999999999995e-05 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:239 3.as0 3:5bb4af71:::smithi09731216-7:head [delete] snapc 4=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 30.762466524000001, 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40857630099999998, 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.785 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014034+0000", 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2500000000000001e-06 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014044+0000", 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1889999999999996e-06 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014067+0000", 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3251000000000002e-05 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114145+0000", 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078193 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114306+0000", 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161044 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114320+0000", 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4044e-05 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317101+0000", 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20278075300000001 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317362+0000", 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026128799999999998 2024-04-18T21:52:24.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.319467+0000", 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021048719999999998 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.326280+0000", 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006813375 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422400+0000", 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096119880000000005 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422608+0000", 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020816199999999999 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:308 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~737280] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:24.791 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.686026448, 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69365002799999997, 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 308 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090475+0000", 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.976e-06 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091768+0000", 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00129317 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091803+0000", 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4502999999999998e-05 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091826+0000", 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3235999999999999e-05 2024-04-18T21:52:24.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270501+0000", 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.178674888 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270680+0000", 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017917599999999999 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.273556+0000", 2024-04-18T21:52:24.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028757610000000001 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281580+0000", 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080240799999999994 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783886+0000", 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50230573000000001 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784122+0000", 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023650799999999999 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:309 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.683945455, 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69216249600000002, 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.797 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 309 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092555+0000", 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093122+0000", 2024-04-18T21:52:24.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056744499999999997 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093146+0000", 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128999999999999e-05 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093165+0000", 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8300000000000001e-05 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275352+0000", 2024-04-18T21:52:24.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18218714899999999 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275632+0000", 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028059100000000001 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.277549+0000", 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001917087 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281584+0000", 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004034513 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784502+0000", 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50291762600000001 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784716+0000", 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021415599999999999 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.802 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:310 3.as0 3:5bb4af71:::smithi09731216-7:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.683304284999998, 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69187687499999995, 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 310 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093196+0000", 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093210+0000", 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3517e-05 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093227+0000", 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7050000000000001e-05 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093243+0000", 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5965999999999999e-05 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279008+0000", 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.185765398 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279287+0000", 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027941400000000002 2024-04-18T21:52:24.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.280700+0000", 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412581 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482172+0000", 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20147153900000001 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784956+0000", 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30278430299999998 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785071+0000", 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115454 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:311 3.as0 3:5bb4af71:::smithi09731216-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:24.808 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.683229333, 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99553992999999996, 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 311 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093271+0000", 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-04-18T21:52:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093278+0000", 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1459999999999999e-06 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093295+0000", 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7408999999999999e-05 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093328+0000", 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2276000000000001e-05 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785901+0000", 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.692573628 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.786181+0000", 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027934999999999999 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.088809+0000", 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30262875700000003 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.396409879, 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30330003799999999, 2024-04-18T21:52:24.813 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380089+0000", 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380090+0000", 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.47e-07 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380096+0000", 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6100000000000002e-06 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380115+0000", 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8986000000000001e-05 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.380129+0000", 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4229e-05 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683350+0000", 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30322019500000003 2024-04-18T21:52:24.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683389+0000", 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9171000000000002e-05 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=14, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.395167345000001, 2024-04-18T21:52:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30209698499999998, 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381331+0000", 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381333+0000", 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.843e-06 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381343+0000", 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8700000000000004e-06 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381368+0000", 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4613e-05 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.381381+0000", 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2857000000000001e-05 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683415+0000", 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30203418999999998 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.683428+0000", 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3611999999999999e-05 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=13, last_complete=41'9, committed=1, applied=1))", 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.293934435000001, 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30158400099999999, 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.821 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482564+0000", 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482566+0000", 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4360000000000001e-06 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482574+0000", 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4700000000000002e-06 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482593+0000", 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9015999999999999e-05 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482608+0000", 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5023e-05 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783615+0000", 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30100726999999999 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784148+0000", 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053278600000000005 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.824 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=14, last_complete=41'10, committed=1, applied=1))", 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.293846064, 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30208896099999999, 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482653+0000", 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5000000000000002e-07 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482657+0000", 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1269999999999996e-06 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482672+0000", 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4688999999999999e-05 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482683+0000", 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1165999999999999e-05 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784263+0000", 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30157952599999999 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784742+0000", 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047880299999999998 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.828 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 41/26 ECSubWriteReply(tid=15, last_complete=41'11, committed=1, applied=1))", 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.293801731999999, 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30306911800000003, 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482697+0000", 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482701+0000", 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1470000000000001e-06 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482712+0000", 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.182e-05 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482719+0000", 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9129999999999997e-06 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784839+0000", 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30211942400000003 2024-04-18T21:52:24.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785766+0000", 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092735400000000003 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:427 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.331061608000001, 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38249087900000001, 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.832 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 427 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445440+0000", 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2930000000000001e-06 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445454+0000", 2024-04-18T21:52:24.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3883000000000001e-05 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445492+0000", 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8090000000000003e-05 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445519+0000", 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6452999999999999e-05 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445801+0000", 2024-04-18T21:52:24.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028185900000000003 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445822+0000", 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1418999999999999e-05 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.550566+0000", 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104743788 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.550788+0000", 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022258600000000001 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.552194+0000", 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001405688 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659810+0000", 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107616082 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.827786+0000", 2024-04-18T21:52:24.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16797619999999999 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.827928+0000", 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000141538 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:428 3.10s0 3:0e754656:::smithi09731216-19:head [rollback 9] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.330924594000001, 2024-04-18T21:52:24.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38329886600000002, 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 428 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:24.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445576+0000", 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.514e-06 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445583+0000", 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6429999999999995e-06 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445626+0000", 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2778999999999998e-05 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445644+0000", 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7597000000000001e-05 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445947+0000", 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000303462 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445957+0000", 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6059999999999992e-06 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554327+0000", 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10837039 2024-04-18T21:52:24.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554582+0000", 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000254906 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.556153+0000", 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015706660000000001 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659815+0000", 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10366275 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.828747+0000", 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16893206199999999 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.828873+0000", 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012549100000000001 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.844 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:429 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.330822897999999, 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51405278700000001, 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 429 2024-04-18T21:52:24.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445677+0000", 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5260000000000001e-06 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445685+0000", 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445703+0000", 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8295999999999999e-05 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445718+0000", 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5383000000000002e-05 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545773+0000", 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100054752 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545785+0000", 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1783000000000001e-05 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557166+0000", 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011381578 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557443+0000", 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027702800000000002 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.558675+0000", 2024-04-18T21:52:24.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012320479999999999 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659884+0000", 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101208162 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959511+0000", 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29962765699999999 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959729+0000", 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021733700000000001 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.4.19:7 3.as0 3:5bb4af71:::smithi09731216-7:head [assert-version v11,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.221785985, 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40581794100000002, 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:24.851 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554716+0000", 2024-04-18T21:52:24.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999999e-06 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554731+0000", 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4722999999999999e-05 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554766+0000", 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5216000000000003e-05 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554789+0000", 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2495999999999999e-05 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.749822+0000", 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19503310099999999 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.749982+0000", 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016067800000000001 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.960531+0000", 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.210548227 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:495 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~737280] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2694594649999997, 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.664354899, 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.855 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 495 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:24.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507043+0000", 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.596e-06 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508389+0000", 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013461460000000001 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508432+0000", 2024-04-18T21:52:24.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3294999999999998e-05 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508456+0000", 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4134e-05 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581521+0000", 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.073064403 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581827+0000", 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030644999999999999 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.586475+0000", 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046472989999999997 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.699953+0000", 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.113478523 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171243+0000", 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47128973499999999 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171394+0000", 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000151318 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:496 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~573440] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2662718120000003, 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76183363900000001, 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 496 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.861 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510229+0000", 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.007e-06 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511015+0000", 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000785792 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511046+0000", 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0873000000000001e-05 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511061+0000", 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5293000000000001e-05 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.691993+0000", 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.180931701 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.692191+0000", 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019870099999999999 2024-04-18T21:52:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.694402+0000", 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022106690000000002 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.701975+0000", 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075735730000000001 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.271924+0000", 2024-04-18T21:52:24.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56994823500000003 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272061+0000", 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013679499999999999 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:497 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~196608] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:24.866 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2652224499999996, 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76114050600000005, 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 497 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511278+0000", 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.387e-06 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511608+0000", 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033003500000000001 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511637+0000", 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9706e-05 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511661+0000", 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3439999999999999e-05 2024-04-18T21:52:24.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696155+0000", 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18449448800000001 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696457+0000", 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000301496 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.698580+0000", 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021231879999999998 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.702633+0000", 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040534079999999997 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272305+0000", 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56967170199999995 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272417+0000", 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111656 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 64/31 ECSubWrite(tid=40, reqid=osd.11.0:40, at_version=64'8, trim_to=0'0, roll_forward_to=62'6))", 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.572245+0000", 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2042536070000001, 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31376227600000001, 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.872 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572245+0000", 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572245+0000", 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572245+0000", 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2e-07 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572283+0000", 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7953000000000001e-05 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572307+0000", 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3626e-05 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572323+0000", 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6367000000000001e-05 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.880773+0000", 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30845004199999998 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.880788+0000", 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4749999999999999e-05 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.885929+0000", 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051404720000000001 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886007+0000", 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8646000000000006e-05 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.12s6 64/54 ECSubWrite(tid=8, reqid=osd.9.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=64'13))", 2024-04-18T21:52:24.876 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.881713+0000", 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.894785342, 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30192024099999998, 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.881713+0000", 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:24.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.881713+0000", 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.881715+0000", 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5999999999999999e-06 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.881736+0000", 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0607000000000001e-05 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.881759+0000", 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3362000000000001e-05 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.881774+0000", 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.537e-05 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.982017+0000", 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100242582 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.982033+0000", 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5656000000000001e-05 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.183548+0000", 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20151525200000001 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.183634+0000", 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5811999999999998e-05 2024-04-18T21:52:24.880 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.881 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:24.881 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:24.881 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:24.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:24.883 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:24.879+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:24.983 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:24.979+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:25.053 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:25.053 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:25.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:25.084 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:25.084+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:25.185 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:25.184+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:25.188 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:25.188 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:25.188 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:25.189 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:25.189 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:25.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:25.286 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:25.284+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:25.331 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 55.529192997999999, 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58281355499999998, 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 1 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798265+0000", 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798268+0000", 2024-04-18T21:52:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5959999999999998e-06 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798283+0000", 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5495999999999999e-05 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798322+0000", 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8899999999999997e-05 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.798347+0000", 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5510000000000001e-05 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886734+0000", 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088386079000000006 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:29.886749+0000", 2024-04-18T21:52:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5019e-05 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.078589+0000", 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19184037900000001 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079052+0000", 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046354399999999998 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.079551+0000", 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049870299999999997 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.083572+0000", 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040212609999999999 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.380910+0000", 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29733748900000001 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381009+0000", 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9025999999999997e-05 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:30.381079+0000", 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9553000000000003e-05 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.592342874000003, 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50698967299999997, 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 34 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735115+0000", 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735116+0000", 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735123+0000", 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3629999999999996e-06 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735156+0000", 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3030000000000001e-05 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735167+0000", 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.13e-05 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939428+0000", 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20426096299999999 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939615+0000", 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018654199999999999 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.939841+0000", 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022621800000000001 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.141747+0000", 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201905318 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.241958+0000", 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100211063 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242041+0000", 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3300000000000005e-05 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242105+0000", 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3895999999999999e-05 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.592267874999997, 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61088117099999995, 2024-04-18T21:52:25.344 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 35 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735191+0000", 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3699999999999999e-07 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735266+0000", 2024-04-18T21:52:25.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4856000000000003e-05 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735279+0000", 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.36e-05 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735288+0000", 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0860000000000008e-06 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242298+0000", 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50700964999999998 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242568+0000", 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027023100000000001 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242974+0000", 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040533399999999998 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246576+0000", 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036026919999999998 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345792+0000", 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099215503999999996 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345953+0000", 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161189 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346071+0000", 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118192 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.592155382000001, 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61108955600000003, 2024-04-18T21:52:25.350 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 36 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.23e-07 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735310+0000", 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7560000000000002e-06 2024-04-18T21:52:25.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735323+0000", 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735337+0000", 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4049999999999999e-05 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244243+0000", 2024-04-18T21:52:25.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50890638099999996 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244422+0000", 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000178495 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244638+0000", 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216562 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.247950+0000", 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033121750000000001 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346232+0000", 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098281564000000002 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346324+0000", 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2042999999999999e-05 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346392+0000", 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8052999999999994e-05 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.592106362999999, 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71224671699999997, 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.356 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 37 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0999999999999998e-07 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735358+0000", 2024-04-18T21:52:25.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2639999999999999e-06 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735367+0000", 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735375+0000", 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5360000000000004e-06 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245345+0000", 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50996983699999998 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245615+0000", 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027018099999999998 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246022+0000", 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000407394 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.248536+0000", 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025132330000000001 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447444+0000", 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198908419 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447532+0000", 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8010000000000006e-05 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447598+0000", 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6383000000000005e-05 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 47.67870585, 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61732750999999997, 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.362 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 117 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648753+0000", 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648758+0000", 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648768+0000", 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.414e-06 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648777+0000", 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.363e-06 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863479+0000", 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214701524 2024-04-18T21:52:25.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863596+0000", 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117039 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863810+0000", 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021416800000000001 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.864846+0000", 2024-04-18T21:52:25.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001036319 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265892+0000", 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40104641899999999 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265995+0000", 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010209600000000001 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.266080+0000", 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5049000000000005e-05 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 44.021794610000001, 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.509317829, 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.368 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 158 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305663+0000", 2024-04-18T21:52:25.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305665+0000", 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5230000000000001e-06 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305786+0000", 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012069 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305805+0000", 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9836000000000001e-05 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305821+0000", 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6030000000000001e-05 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307593+0000", 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017715299999999999 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.307838+0000", 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024551800000000002 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.308174+0000", 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033593400000000003 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413731+0000", 2024-04-18T21:52:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105556865 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814825+0000", 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40109329500000002 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814911+0000", 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6138999999999996e-05 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.814981+0000", 2024-04-18T21:52:25.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0469000000000006e-05 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 44.021609554000001, 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50938204099999995, 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.374 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 159 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305848+0000", 2024-04-18T21:52:25.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305849+0000", 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0559999999999999e-06 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305856+0000", 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9600000000000003e-06 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305867+0000", 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0910000000000001e-05 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305879+0000", 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.147e-05 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309071+0000", 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031920859999999998 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309303+0000", 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023236200000000001 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.309671+0000", 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036798000000000001 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416908+0000", 2024-04-18T21:52:25.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107236578 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815085+0000", 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39817670700000002 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815165+0000", 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0196000000000003e-05 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815230+0000", 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5735999999999995e-05 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 44.021566251000003, 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50958771300000005, 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 160 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305892+0000", 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305898+0000", 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1359999999999996e-06 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305907+0000", 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5300000000000002e-06 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305915+0000", 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1869999999999997e-06 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310667+0000", 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047516800000000003 2024-04-18T21:52:25.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.310895+0000", 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000227875 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.311257+0000", 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000361737 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416933+0000", 2024-04-18T21:52:25.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10567686699999999 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815344+0000", 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39841079899999998 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815421+0000", 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7229000000000002e-05 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.815479+0000", 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7973000000000002e-05 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 44.021531588999999, 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60955713600000005, 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.386 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 161 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305927+0000", 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306056+0000", 2024-04-18T21:52:25.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012895300000000001 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306068+0000", 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306077+0000", 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9430000000000006e-06 2024-04-18T21:52:25.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412179+0000", 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106101709 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412267+0000", 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8251999999999995e-05 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412388+0000", 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121136 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420472+0000", 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080833510000000008 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915141+0000", 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49466944899999998 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915356+0000", 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215305 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915484+0000", 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012713500000000001 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 44.021367765999997, 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61031149600000001, 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 162 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.393 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306091+0000", 2024-04-18T21:52:25.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306097+0000", 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6300000000000003e-06 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306106+0000", 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2199999999999998e-06 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306116+0000", 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8870000000000004e-06 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413869+0000", 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107753534 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414135+0000", 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026615200000000002 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414518+0000", 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038217299999999997 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420492+0000", 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059740879999999998 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915940+0000", 2024-04-18T21:52:25.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49544827600000002 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916143+0000", 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000202752 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916402+0000", 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025901100000000002 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.398 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 44.021331396000001, 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61064147300000005, 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.399 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 163 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.06e-07 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306136+0000", 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6999999999999997e-06 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306146+0000", 2024-04-18T21:52:25.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8439999999999993e-06 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306154+0000", 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415593+0000", 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109439392 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415833+0000", 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000240101 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416223+0000", 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000389804 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420508+0000", 2024-04-18T21:52:25.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042851699999999996 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916537+0000", 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49602917499999999 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916669+0000", 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179899999999999 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916768+0000", 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8879e-05 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 44.021292862999999, 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61096294100000004, 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.405 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 164 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306166+0000", 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306275+0000", 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109749 2024-04-18T21:52:25.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306287+0000", 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1182999999999999e-05 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306298+0000", 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1487e-05 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417044+0000", 2024-04-18T21:52:25.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110745818 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417282+0000", 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000237712 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417665+0000", 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000383194 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420523+0000", 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028578010000000001 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916880+0000", 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49635731900000002 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917054+0000", 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017436200000000001 2024-04-18T21:52:25.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917128+0000", 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3693000000000001e-05 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 44.021146868000002, 2024-04-18T21:52:25.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70940268200000001, 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 165 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:25.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306312+0000", 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306317+0000", 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0699999999999997e-06 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306327+0000", 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7030000000000001e-06 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306349+0000", 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2160000000000001e-05 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419233+0000", 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112884159 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419499+0000", 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265865 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419851+0000", 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035253999999999998 2024-04-18T21:52:25.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.421040+0000", 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011886380000000001 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015628+0000", 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59458825299999996 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015677+0000", 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8470000000000002e-05 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015714+0000", 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7123999999999999e-05 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:438 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~622592] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:25.417 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.845383364, 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60190392599999998, 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 438 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482077+0000", 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3429999999999998e-06 2024-04-18T21:52:25.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482901+0000", 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082364099999999996 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482933+0000", 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2128999999999997e-05 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482949+0000", 2024-04-18T21:52:25.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5872999999999999e-05 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676460+0000", 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19351131799999999 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676647+0000", 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018723700000000001 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.678595+0000", 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00194812 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.782086+0000", 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10349056299999999 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083745+0000", 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30165924199999999 2024-04-18T21:52:25.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083979+0000", 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023346 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:439 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~147456] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.843765878999999, 2024-04-18T21:52:25.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70156516499999999, 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 439 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483694+0000", 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2129999999999998e-06 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483902+0000", 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020762600000000001 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483934+0000", 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2140000000000001e-05 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483950+0000", 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5792999999999999e-05 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779688+0000", 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29573833799999999 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779887+0000", 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019900000000000001 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.781116+0000", 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001229071 2024-04-18T21:52:25.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.882567+0000", 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101451022 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185116+0000", 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.302548704 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185257+0000", 2024-04-18T21:52:25.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014125800000000001 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:440 3.fs0 3:fcd1a062:::smithi09731216-16:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.843482004, 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.10743576, 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.429 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 440 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483977+0000", 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7000000000000003e-07 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483982+0000", 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4e-06 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483994+0000", 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.134e-05 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484003+0000", 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1169999999999993e-06 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186403+0000", 2024-04-18T21:52:25.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70240069199999999 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186624+0000", 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000220523 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.187505+0000", 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088087199999999999 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188877+0000", 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001371723 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591138+0000", 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40226145699999999 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591412+0000", 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027366599999999999 2024-04-18T21:52:25.434 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:441 3.fs0 3:fcd1a062:::smithi09731216-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.843442761, 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.413060682, 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:25.435 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 441 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:25.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484016+0000", 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8299999999999996e-07 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484019+0000", 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3069999999999998e-06 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484038+0000", 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978999999999999e-05 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484047+0000", 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5939999999999994e-06 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188125+0000", 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70407843599999997 2024-04-18T21:52:25.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188355+0000", 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022941899999999999 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692687+0000", 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50433201000000005 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692919+0000", 2024-04-18T21:52:25.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023258299999999999 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.897076+0000", 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20415657100000001 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:493 3.13s0 3:c8a99f7f:::smithi09731216-31:6 [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.495634+0000", 2024-04-18T21:52:25.440 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.83182372, 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59013068499999999, 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 493 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495634+0000", 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495634+0000", 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495636+0000", 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.657e-06 2024-04-18T21:52:25.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495645+0000", 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.85e-06 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495666+0000", 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1413e-05 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495684+0000", 2024-04-18T21:52:25.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8187000000000001e-05 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.582898+0000", 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087213395999999999 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.583217+0000", 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000319609 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.884673+0000", 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30145561399999998 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.884887+0000", 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021448100000000001 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.085765+0000", 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200877478 2024-04-18T21:52:25.445 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:494 3.13s0 3:c8a99f7f:::smithi09731216-31:6 [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.8317257940000005, 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59079312500000003, 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:25.446 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 494 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:25.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495734+0000", 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4929999999999999e-06 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495743+0000", 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5629999999999998e-06 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495768+0000", 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5147000000000002e-05 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495784+0000", 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6019e-05 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.596290+0000", 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100505504 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.596489+0000", 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019867499999999999 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886268+0000", 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.28977927399999998 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886434+0000", 2024-04-18T21:52:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000166288 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.086525+0000", 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20009116199999999 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:25.451 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:25.452 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is unset 2024-04-18T21:52:25.453 INFO:tasks.rados.rados.0.smithi097.stdout:212: done (3 left) 2024-04-18T21:52:25.453 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 19 current snap is 18 2024-04-18T21:52:25.453 INFO:tasks.rados.rados.0.smithi097.stdout:213: seq_num 93 ranges {2834432=802816,3637248=638976,4276224=131072} 2024-04-18T21:52:25.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:25.455 INFO:tasks.rados.rados.0.smithi097.stdout:213: writing smithi09731216-19 from 2834432 to 3637248 tid 1 2024-04-18T21:52:25.455 INFO:tasks.rados.rados.0.smithi097.stdout:213: writing smithi09731216-19 from 3637248 to 4276224 tid 2 2024-04-18T21:52:25.455 INFO:tasks.rados.rados.0.smithi097.stdout:213: writing smithi09731216-19 from 4276224 to 4407296 tid 3 2024-04-18T21:52:25.455 INFO:tasks.rados.rados.0.smithi097.stdout:214: copy_from oid 38 from oid 46 current snap is 18 2024-04-18T21:52:25.456 INFO:tasks.rados.rados.0.smithi097.stdout:208: expect (ObjNum 87 snap 15 seq_num 87) 2024-04-18T21:52:25.456 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:25.452+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:25.458 INFO:tasks.rados.rados.0.smithi097.stdout:210: finishing write tid 1 to smithi09731216-20 2024-04-18T21:52:25.458 INFO:tasks.rados.rados.0.smithi097.stdout:210: finishing write tid 2 to smithi09731216-20 2024-04-18T21:52:25.458 INFO:tasks.rados.rados.0.smithi097.stdout:210: finishing write tid 3 to smithi09731216-20 2024-04-18T21:52:25.458 INFO:tasks.rados.rados.0.smithi097.stdout:209: finishing write tid 1 to smithi09731216-37 2024-04-18T21:52:25.458 INFO:tasks.rados.rados.0.smithi097.stdout:210: finishing write tid 4 to smithi09731216-20 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:209: finishing write tid 2 to smithi09731216-37 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:209: finishing write tid 3 to smithi09731216-37 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:209: finishing write tid 4 to smithi09731216-37 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:210: finishing write tid 5 to smithi09731216-20 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:210: oid 20 updating version 0 to 112 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:210: oid 20 version 112 is already newer than 109 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:210: oid 20 version 112 is already newer than 111 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:210: oid 20 version 112 is already newer than 110 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 20 v 112 (ObjNum 92 snap 18 seq_num 92) dirty exists 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:210: left oid 20 (ObjNum 92 snap 18 seq_num 92) 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:208: done (4 left) 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:210: done (3 left) 2024-04-18T21:52:25.459 INFO:tasks.rados.rados.0.smithi097.stdout:215: snap_remove snap 16 2024-04-18T21:52:25.553 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:25.551+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:25.591 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:25.592 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:25.592 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:25.592 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:25.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:25.654 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:25.651+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:25.730 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:25.731 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:25.731 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:25.731 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:25.731 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:25.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:25.792 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:25.792+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:25.855 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:25.851+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:25.869 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:25.869 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:25.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:25.869 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:25.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.869 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:25.869 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 39.302684837000001, 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201960743, 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566449+0000", 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.77e-06 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566495+0000", 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5488999999999998e-05 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566536+0000", 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0756000000000001e-05 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.666638+0000", 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102466 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766859+0000", 2024-04-18T21:52:25.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10022078500000001 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766875+0000", 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5806000000000001e-05 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768312+0000", 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014373389999999999 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768408+0000", 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6331999999999999e-05 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:118 3.17s0 3:eda737c1:::smithi09731216-40:head [append 0~671744,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 38.264441607999998, 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1801127490000001, 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.874 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 118 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604693+0000", 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2970000000000002e-06 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.605995+0000", 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001302276 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606021+0000", 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5320000000000002e-05 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606036+0000", 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5133000000000001e-05 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606232+0000", 2024-04-18T21:52:25.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000196672 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606237+0000", 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7199999999999997e-06 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484281+0000", 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.87804420999999999 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:25.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484592+0000", 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031024299999999998 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.487837+0000", 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003245571 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490891+0000", 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030539719999999998 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784613+0000", 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.293722237 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784804+0000", 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019009800000000001 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.880 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:119 3.17s0 3:eda737c1:::smithi09731216-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 38.263059759000001, 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.179191986, 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 119 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606074+0000", 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.257e-06 2024-04-18T21:52:25.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606080+0000", 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0100000000000001e-06 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606095+0000", 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5413e-05 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606104+0000", 2024-04-18T21:52:25.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0899999999999994e-06 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606286+0000", 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000181552 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606287+0000", 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.026e-06 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493720+0000", 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88743291700000004 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493872+0000", 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015190200000000001 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495103+0000", 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00123149 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.496855+0000", 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017518189999999999 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785096+0000", 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.28824129900000001 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785265+0000", 2024-04-18T21:52:25.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016821100000000001 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:120 3.17s0 3:eda737c1:::smithi09731216-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 38.262990629000001, 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4107226310000001, 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:25.887 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 120 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606143+0000", 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.16e-07 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606148+0000", 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4569999999999996e-06 2024-04-18T21:52:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606163+0000", 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503e-05 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606175+0000", 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2036e-05 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706293+0000", 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100117875 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706297+0000", 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.126e-06 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495621+0000", 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78932349599999996 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495757+0000", 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013642600000000001 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785995+0000", 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29023745499999998 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.786127+0000", 2024-04-18T21:52:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013212200000000001 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.016864+0000", 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.23073769199999999 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.17s0 32/31 ECSubRead(tid=3, to_read={3:eda737c1:::smithi09731216-40:head=0,4096,0}, subchunks={3:eda737c1:::smithi09731216-40:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:25.893 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.081949627999997, 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.210779366, 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787250+0000", 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7152999999999999e-05 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887353+0000", 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102647 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997704+0000", 2024-04-18T21:52:25.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110351217 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997962+0000", 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000258349 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.17s0 32/31 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.072654624000002, 2024-04-18T21:52:25.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200936119, 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796479+0000", 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8999999999999995e-07 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796491+0000", 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2065000000000001e-05 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796502+0000", 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.116e-05 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796509+0000", 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7170000000000008e-06 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997338+0000", 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20082881799999999 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997414+0000", 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5568999999999994e-05 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.072487903999999, 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200846881, 2024-04-18T21:52:25.901 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796644+0000", 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796645+0000", 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8999999999999998e-07 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796647+0000", 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5239999999999999e-06 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796655+0000", 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796660+0000", 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4879999999999998e-06 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997463+0000", 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20080266799999999 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997491+0000", 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8192e-05 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 32/31 ECSubWriteReply(tid=2, last_complete=32'2, committed=1, applied=1))", 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.071967352999998, 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20052022, 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.905 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797165+0000", 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797166+0000", 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5300000000000004e-07 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797170+0000", 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.233e-06 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797183+0000", 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3454e-05 2024-04-18T21:52:25.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797191+0000", 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5769999999999996e-06 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997659+0000", 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20046852900000001 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997685+0000", 2024-04-18T21:52:25.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5774000000000001e-05 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/31 ECSubWrite(tid=28, reqid=client.4509.0:304, at_version=41'12, trim_to=0'0, roll_forward_to=32'8))", 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.699765061000001, 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201742379, 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169369+0000", 2024-04-18T21:52:25.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6339999999999999e-06 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169413+0000", 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3860999999999999e-05 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169487+0000", 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4578000000000003e-05 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.269583+0000", 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100095613 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369850+0000", 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10026700299999999 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369869+0000", 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9114e-05 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371022+0000", 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001152943 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371110+0000", 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7633000000000006e-05 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/26 ECSubWrite(tid=56, reqid=osd.10.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'25))", 2024-04-18T21:52:25.913 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.083689036999999, 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201706881, 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785445+0000", 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7999999999999999e-06 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785499+0000", 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3730999999999998e-05 2024-04-18T21:52:25.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785543+0000", 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3961000000000001e-05 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785568+0000", 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.561e-05 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.985970+0000", 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200401511 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.986000+0000", 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0224000000000001e-05 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987085+0000", 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001085145 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987150+0000", 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4899000000000003e-05 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.17s0,GRANT,e:42,from:10(3),reservation_nonce:10)", 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.588544006999999, 2024-04-18T21:52:25.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200838461, 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280588+0000", 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280590+0000", 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.587e-06 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280598+0000", 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5450000000000003e-06 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.280627+0000", 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8141e-05 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.380804+0000", 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10017703 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481049+0000", 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10024543 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481088+0000", 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8720999999999997e-05 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.481427+0000", 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000339007 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.922 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 53/51 ECSubWrite(tid=2, reqid=client.4509.0:390, at_version=0'0, trim_to=0'0, roll_forward_to=53'7))", 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 16.170509174999999, 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201345094, 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698625+0000", 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2869999999999999e-06 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698665+0000", 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9743000000000002e-05 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698719+0000", 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4228999999999999e-05 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.798874+0000", 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100154506 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.799005+0000", 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013080200000000001 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.799030+0000", 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5058e-05 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.899930+0000", 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100900189 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.899968+0000", 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8279999999999999e-05 2024-04-18T21:52:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.2s5 58/31 ECSubWrite(tid=87, reqid=client.4509.0:422, at_version=58'6, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 11.210946496, 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201964437, 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658187+0000", 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5069999999999999e-06 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658288+0000", 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010070500000000001 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658315+0000", 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7348e-05 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658331+0000", 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5559000000000001e-05 2024-04-18T21:52:25.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.758532+0000", 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100200734 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.758549+0000", 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6823000000000001e-05 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860062+0000", 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101513567 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860150+0000", 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8194000000000003e-05 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.2s5 58/31 ECSubWrite(tid=88, reqid=client.4509.0:423, at_version=58'7, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:25.931 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 11.210762237999999, 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20184664099999999, 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658371+0000", 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5400000000000001e-07 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658390+0000", 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8907e-05 2024-04-18T21:52:25.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658404+0000", 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3974999999999999e-05 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658414+0000", 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0156000000000001e-05 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.759248+0000", 2024-04-18T21:52:25.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10083389299999999 2024-04-18T21:52:25.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.935 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.935 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.759263+0000", 2024-04-18T21:52:25.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5101e-05 2024-04-18T21:52:25.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860159+0000", 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10089624599999999 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860217+0000", 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7809000000000002e-05 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.fs7 62/26 ECSubWrite(tid=203, reqid=osd.7.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=62'26))", 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.2929257789999999, 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20165132299999999, 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576209+0000", 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9300000000000002e-06 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576234+0000", 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5287999999999999e-05 2024-04-18T21:52:25.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576260+0000", 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5808000000000001e-05 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.676352+0000", 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10009247 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.776560+0000", 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100207532 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.776579+0000", 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8936e-05 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.777791+0000", 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00121206 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.777858+0000", 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7298999999999994e-05 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 64/26 ECSubWrite(tid=97, reqid=osd.5.0:97, at_version=64'55, trim_to=0'0, roll_forward_to=62'51))", 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.2709479220000004, 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.202670817, 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.943 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598185+0000", 2024-04-18T21:52:25.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598203+0000", 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8168e-05 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598214+0000", 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0509999999999999e-05 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.698303+0000", 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100089224 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.798517+0000", 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100213922 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.798550+0000", 2024-04-18T21:52:25.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3120999999999997e-05 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.800786+0000", 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002236334 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.800855+0000", 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8966000000000003e-05 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/54 ECSubWrite(tid=8, reqid=osd.9.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=64'13))", 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.9861205159999997, 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30101452299999998, 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883014+0000", 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3800000000000001e-06 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883043+0000", 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8924999999999998e-05 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883085+0000", 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1674000000000001e-05 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883109+0000", 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4345000000000002e-05 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.083392+0000", 2024-04-18T21:52:25.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200282457 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.083411+0000", 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9156000000000001e-05 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.183969+0000", 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100558567 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.184026+0000", 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7018999999999997e-05 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 64/26 ECSubWrite(tid=99, reqid=osd.5.0:97, at_version=0'0, trim_to=0'0, roll_forward_to=64'55))", 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.9633749800000002, 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201502976, 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.952 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905758+0000", 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.91e-07 2024-04-18T21:52:25.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905764+0000", 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1399999999999997e-06 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905769+0000", 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5029999999999997e-06 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905776+0000", 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6989999999999997e-06 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.106403+0000", 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20062649299999999 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.106420+0000", 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7187999999999999e-05 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.107159+0000", 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00073935799999999996 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.107260+0000", 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101304 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es4 68/26 ECSubWrite(tid=58, reqid=osd.2.0:58, at_version=68'44, trim_to=0'0, roll_forward_to=66'40))", 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:25.956 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.487057627, 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.205606013, 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382077+0000", 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.863e-06 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382131+0000", 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.465e-05 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382164+0000", 2024-04-18T21:52:25.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3081999999999999e-05 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.482269+0000", 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10010452 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.582575+0000", 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100306485 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.582604+0000", 2024-04-18T21:52:25.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8462e-05 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.587585+0000", 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049807999999999996 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.587681+0000", 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6150999999999995e-05 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s3 68/58 ECSubWrite(tid=215, reqid=osd.7.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.948862757, 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20176354299999999, 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920271+0000", 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8530000000000001e-06 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920293+0000", 2024-04-18T21:52:25.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1926e-05 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920324+0000", 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1047999999999998e-05 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.020468+0000", 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100143498 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.120740+0000", 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100271761 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.120767+0000", 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7268999999999999e-05 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.121955+0000", 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011884599999999999 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:25.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.122033+0000", 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7727999999999998e-05 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:25.965 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:25.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:25.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:25.968 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:25.963+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:26.068 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:26.063+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:26.113 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:26.113 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:26.130 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:26.130 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:26.137 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:26.137 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:26.137 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:26.148 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:26.148 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:26.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:26.165 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:26.165 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:26.185 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:26.185 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:26.200 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:26.200+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:26.206 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:26.206 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:26.269 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:26.267+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:26.274 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:26.274 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:26.274 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:26.274 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:26.274 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:26.289 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:26.328 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:26.328 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:26.353 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:26.353 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:26.368 INFO:tasks.rados.rados.0.smithi097.stdout:215: done (3 left) 2024-04-18T21:52:26.368 INFO:tasks.rados.rados.0.smithi097.stdout:216: rmattr oid 8 current snap is 18 2024-04-18T21:52:26.368 INFO:tasks.rados.rados.0.smithi097.stdout:216: done (3 left) 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:217: rmattr oid 24 current snap is 18 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:217: done (3 left) 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:218: rmattr oid 4 current snap is 18 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:218: done (3 left) 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:219: read oid 1 snap -1 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:219: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:220: read oid 6 snap -1 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:220: expect (ObjNum 87 snap 15 seq_num 87) 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:221: setattr oid 35 current snap is 18 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:209: finishing write tid 5 to smithi09731216-37 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:209: oid 37 updating version 0 to 5 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:209: oid 37 updating version 5 to 6 2024-04-18T21:52:26.369 INFO:tasks.rados.rados.0.smithi097.stdout:209: oid 37 updating version 6 to 8 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:209: oid 37 version 8 is already newer than 7 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 37 v 8 (ObjNum 91 snap 18 seq_num 91) dirty exists 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:209: left oid 37 (ObjNum 91 snap 18 seq_num 91) 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:214: finishing copy_from racing read to smithi09731216-38 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: finishing write tid 1 to smithi09731216-19 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: finishing write tid 2 to smithi09731216-19 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: finishing write tid 3 to smithi09731216-19 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: finishing write tid 4 to smithi09731216-19 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: finishing write tid 5 to smithi09731216-19 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: oid 19 updating version 0 to 42 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: oid 19 updating version 42 to 43 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: oid 19 version 43 is already newer than 41 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:213: oid 19 updating version 43 to 44 2024-04-18T21:52:26.370 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 19 v 44 (ObjNum 93 snap 18 seq_num 93) dirty exists 2024-04-18T21:52:26.371 INFO:tasks.rados.rados.0.smithi097.stdout:213: left oid 19 (ObjNum 93 snap 18 seq_num 93) 2024-04-18T21:52:26.371 INFO:tasks.rados.rados.0.smithi097.stdout:209: done (5 left) 2024-04-18T21:52:26.371 INFO:tasks.rados.rados.0.smithi097.stdout:213: done (4 left) 2024-04-18T21:52:26.372 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 34 current snap is 18 2024-04-18T21:52:26.372 INFO:tasks.rados.rados.0.smithi097.stdout:222: seq_num 94 ranges {606208=425984,1032192=786432,1818624=376832} 2024-04-18T21:52:26.372 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:26.367+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:26.379 INFO:tasks.rados.rados.0.smithi097.stdout:222: writing smithi09731216-34 from 606208 to 1032192 tid 1 2024-04-18T21:52:26.383 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:26.383 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:26.386 INFO:tasks.rados.rados.0.smithi097.stdout:222: writing smithi09731216-34 from 1032192 to 1818624 tid 2 2024-04-18T21:52:26.390 INFO:tasks.rados.rados.0.smithi097.stdout:222: writing smithi09731216-34 from 1818624 to 2195456 tid 3 2024-04-18T21:52:26.390 INFO:tasks.rados.rados.0.smithi097.stdout:223: copy_from oid 7 from oid 27 current snap is 18 2024-04-18T21:52:26.390 INFO:tasks.rados.rados.0.smithi097.stdout:224: snap_create 2024-04-18T21:52:26.405 INFO:tasks.rados.rados.0.smithi097.stdout:214: finishing copy_from to smithi09731216-38 2024-04-18T21:52:26.406 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 38 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-18T21:52:26.409 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:26.409 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:26.459 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:26.459 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:26.472 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:26.472+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:26.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:26.494 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:26.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:52:26.494 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:26.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-04-18T21:52:26.505 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:26.506 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:26.529 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:26.529 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:26.573 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:26.571+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:26.598 INFO:tasks.rados.rados.0.smithi097.stdout:222: finishing write tid 1 to smithi09731216-34 2024-04-18T21:52:26.640 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:26.640 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:26.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:26.700 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:26.700+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:26.770 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:26.770 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:26.775 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:26.771+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:26.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:26.805 INFO:tasks.rados.rados.0.smithi097.stdout:222: finishing write tid 2 to smithi09731216-34 2024-04-18T21:52:26.806 INFO:tasks.rados.rados.0.smithi097.stdout:222: finishing write tid 3 to smithi09731216-34 2024-04-18T21:52:26.806 INFO:tasks.rados.rados.0.smithi097.stdout:222: finishing write tid 4 to smithi09731216-34 2024-04-18T21:52:26.811 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-18T21:52:26.807+0000 7f0412e7b640 -1 mon.a@0(leader).osd e70 definitely_dead 0 2024-04-18T21:52:26.873 INFO:tasks.rados.rados.0.smithi097.stdout:219: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:52:26.875 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:26.871+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:26.948 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:26.948 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:26.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:27.011 INFO:tasks.rados.rados.0.smithi097.stdout:222: finishing write tid 5 to smithi09731216-34 2024-04-18T21:52:27.011 INFO:tasks.rados.rados.0.smithi097.stdout:222: oid 34 updating version 0 to 57 2024-04-18T21:52:27.011 INFO:tasks.rados.rados.0.smithi097.stdout:222: oid 34 updating version 57 to 60 2024-04-18T21:52:27.011 INFO:tasks.rados.rados.0.smithi097.stdout:222: oid 34 version 60 is already newer than 58 2024-04-18T21:52:27.011 INFO:tasks.rados.rados.0.smithi097.stdout:222: oid 34 version 60 is already newer than 59 2024-04-18T21:52:27.011 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 34 v 60 (ObjNum 94 snap 18 seq_num 94) dirty exists 2024-04-18T21:52:27.011 INFO:tasks.rados.rados.0.smithi097.stdout:222: left oid 34 (ObjNum 94 snap 18 seq_num 94) 2024-04-18T21:52:27.012 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:27.012+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:27.077 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:27.075+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:27.084 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:27.084 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:27.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:27.119 INFO:tasks.rados.rados.0.smithi097.stdout:223: finishing copy_from racing read to smithi09731216-7 2024-04-18T21:52:27.178 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:27.175+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:27.266 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:27.266 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:27.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:27.279 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:27.275+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:27.321 INFO:tasks.rados.rados.0.smithi097.stdout:223: finishing copy_from to smithi09731216-7 2024-04-18T21:52:27.321 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 7 v 13 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-18T21:52:27.368 INFO:teuthology.orchestra.run.smithi097.stderr:marked out osd.0. 2024-04-18T21:52:27.372 INFO:tasks.rados.rados.0.smithi097.stdout:214: done (6 left) 2024-04-18T21:52:27.372 INFO:tasks.rados.rados.0.smithi097.stdout:219: done (5 left) 2024-04-18T21:52:27.373 INFO:tasks.rados.rados.0.smithi097.stdout:222: done (4 left) 2024-04-18T21:52:27.373 INFO:tasks.rados.rados.0.smithi097.stdout:223: done (3 left) 2024-04-18T21:52:27.373 INFO:tasks.rados.rados.0.smithi097.stdout:224: done (2 left) 2024-04-18T21:52:27.373 INFO:tasks.rados.rados.0.smithi097.stdout:225: delete oid 36 current snap is 19 2024-04-18T21:52:27.381 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:27.379+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:27.401 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:27.402 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:27.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:27.423 INFO:tasks.rados.rados.0.smithi097.stdout:220: expect (ObjNum 87 snap 15 seq_num 87) 2024-04-18T21:52:27.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-18T21:52:27.491 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:27.487+0000 7efd0e2e6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:27.513 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:27.511+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:27.547 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:27.547 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:27.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:27.609 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:27.607+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:27.681 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:27.675+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:27.684 INFO:tasks.rados.rados.0.smithi097.stdout:220: done (2 left) 2024-04-18T21:52:27.684 INFO:tasks.rados.rados.0.smithi097.stdout:225: done (1 left) 2024-04-18T21:52:27.684 INFO:tasks.rados.rados.0.smithi097.stdout:226: delete oid 3 current snap is 19 2024-04-18T21:52:27.731 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:27.731 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:27.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:27.781 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:27.779+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:27.882 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:27.879+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:27.909 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:27.909 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:27.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:27.983 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:27.980+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:28.045 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:28.046 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:28.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:28.084 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:28.084+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:28.181 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:28.182 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:28.185 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:28.183+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:28.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:28.285 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:28.283+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:28.361 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:28.361 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:28.371 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is set 2024-04-18T21:52:28.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:28.393 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:28.383+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:28.487 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:28.483+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:28.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:28.565 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:28.565 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:28.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:28.621 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:28.619+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:28.688 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:28.683+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:28.694 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:28.711 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:28.711 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:28.728 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:28.728 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:28.738 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:28.739 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:28.739 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:28.739 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:28.739 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:28.745 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:28.745 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:28.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:28.766 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:28.766 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:28.784 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:28.783+0000 7efd122ee640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:28.787 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:28.787 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:28.789 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:28.787+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:28.808 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:28.808 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:28.828 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:28.829 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:28.850 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:28.851 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:28.872 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:28.872 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:28.883 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:28.883 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:28.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:83 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.831050503, 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28825372900000001, 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 83 2024-04-18T21:52:28.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2e-07 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050828+0000", 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3929999999999998e-06 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050832+0000", 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4909999999999999e-06 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.886 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050836+0000", 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5219999999999999e-06 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052455+0000", 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016189710000000001 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052552+0000", 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7213999999999995e-05 2024-04-18T21:52:28.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052983+0000", 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043119300000000002 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138342+0000", 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.085359090999999998 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.338942+0000", 2024-04-18T21:52:28.888 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200600308 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339078+0000", 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136026 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:84 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.831021845000002, 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59192865299999997, 2024-04-18T21:52:28.889 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 84 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050854+0000", 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1399999999999998e-07 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050855+0000", 2024-04-18T21:52:28.891 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5719999999999999e-06 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050860+0000", 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050864+0000", 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5750000000000001e-06 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138083+0000", 2024-04-18T21:52:28.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087218981000000001 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138226+0000", 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000143545 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339757+0000", 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20153072 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.893 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339856+0000", 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9247999999999996e-05 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642782+0000", 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30292579400000003 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.es0 30/26 ECSubRead(tid=10, to_read={3:77cf8780:::smithi09731216-28:head=0,4096,0}, subchunks={3:77cf8780:::smithi09731216-28:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:28.894 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.541861271999998, 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218228800000002, 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.895 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.896 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340093+0000", 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8665000000000007e-05 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440291+0000", 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100198545 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641845+0000", 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20155352000000001 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642196+0000", 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035155800000000002 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.cs7 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.118320093000001, 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30181939000000002, 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.898 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.899 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763557+0000", 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.776e-06 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763600+0000", 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3120000000000001e-05 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763635+0000", 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5199e-05 2024-04-18T21:52:28.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.863748+0000", 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100112399 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964004+0000", 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10025664500000001 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964030+0000", 2024-04-18T21:52:28.901 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5299999999999998e-05 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065244+0000", 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101214472 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065374+0000", 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000130479 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.902 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=19, last_complete=35'11, committed=1, applied=1))", 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.603664336000001, 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22495391000000001, 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:28.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278211+0000", 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278212+0000", 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.004e-06 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278218+0000", 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7490000000000004e-06 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.278233+0000", 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5815999999999999e-05 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378308+0000", 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007474600000001 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.502895+0000", 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.124586697 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.503165+0000", 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026989800000000002 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=20, last_complete=35'12, committed=1, applied=1))", 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.503538302999999, 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225695955, 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378338+0000", 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3499999999999996e-07 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378343+0000", 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0669999999999999e-06 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378356+0000", 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3264e-05 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378367+0000", 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1069e-05 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.603689+0000", 2024-04-18T21:52:28.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22532233900000001 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604033+0000", 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034358099999999998 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=21, last_complete=35'13, committed=1, applied=1))", 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:28.910 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.503497948000003, 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22690856100000001, 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:28.911 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378378+0000", 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378380+0000", 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7190000000000001e-06 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378389+0000", 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378395+0000", 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5210000000000002e-06 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604185+0000", 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22579012100000001 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.605286+0000", 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011008979999999999 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:230 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 35.868606599000003, 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.312981385, 2024-04-18T21:52:28.914 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 230 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013270+0000", 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5300000000000002e-07 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013275+0000", 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-04-18T21:52:28.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013291+0000", 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5466000000000001e-05 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013303+0000", 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2e-05 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013750+0000", 2024-04-18T21:52:28.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044700300000000002 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013767+0000", 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7263999999999999e-05 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225043+0000", 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21127590299999999 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225199+0000", 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155716 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.326250+0000", 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10105134 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:315 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.789098522, 2024-04-18T21:52:28.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70415003200000004, 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 315 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092778+0000", 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.725e-06 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092785+0000", 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7950000000000002e-06 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092802+0000", 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6852999999999999e-05 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092810+0000", 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7950000000000008e-06 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152681+0000", 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059871316000000001 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152755+0000", 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4179999999999998e-05 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153317+0000", 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00056207900000000005 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253434+0000", 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100116332 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796715+0000", 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54328157899999996 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796927+0000", 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021137799999999999 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:316 3.es0 3:7b3c3586:::smithi09731216-12:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:28.924 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.789049777999999, 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70474535999999999, 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 316 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092826+0000", 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5999999999999999e-07 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092827+0000", 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714e-06 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092833+0000", 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0989999999999998e-06 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092836+0000", 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.056e-06 2024-04-18T21:52:28.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153954+0000", 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061118211999999998 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154059+0000", 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010548399999999999 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154712+0000", 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065292300000000002 2024-04-18T21:52:28.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253436+0000", 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098723985 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797355+0000", 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.543919245 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797571+0000", 2024-04-18T21:52:28.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021528199999999999 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:317 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.789009374999999, 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70519442399999999, 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:28.930 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 317 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:28.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000002e-07 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092868+0000", 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1440000000000001e-06 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092873+0000", 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0860000000000001e-06 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192949+0000", 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007585099999999 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252838+0000", 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059889100000000001 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252927+0000", 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8651000000000002e-05 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253260+0000", 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033289600000000001 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.353758+0000", 2024-04-18T21:52:28.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100498198 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797947+0000", 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.444189102 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.798060+0000", 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113109 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.935 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=6, reqid=client.4509.0:267, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.176409879000001, 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.305911983, 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705468+0000", 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4229999999999999e-06 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705683+0000", 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021499900000000001 2024-04-18T21:52:28.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705719+0000", 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6758999999999998e-05 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805843+0000", 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10012386299999999 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806004+0000", 2024-04-18T21:52:28.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160722 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806040+0000", 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6335999999999999e-05 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011325+0000", 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20528459499999999 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011377+0000", 2024-04-18T21:52:28.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2286e-05 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=8, reqid=client.4509.0:268, at_version=41'15, trim_to=0'0, roll_forward_to=41'12))", 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.075886832999998, 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20598443299999999, 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805988+0000", 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805989+0000", 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5700000000000002e-07 2024-04-18T21:52:28.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806154+0000", 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000165575 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806171+0000", 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6946e-05 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806183+0000", 2024-04-18T21:52:28.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1394e-05 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.009717+0000", 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203534044 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.009751+0000", 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.455e-05 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011896+0000", 2024-04-18T21:52:28.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002144721 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011973+0000", 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6645999999999998e-05 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=9, reqid=client.4509.0:269, at_version=41'16, trim_to=0'0, roll_forward_to=41'12))", 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.075673448, 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.206203308, 2024-04-18T21:52:28.944 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806202+0000", 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.37e-07 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806231+0000", 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8955999999999999e-05 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806247+0000", 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5627000000000001e-05 2024-04-18T21:52:28.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.906369+0000", 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100122086 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011040+0000", 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104670897 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011064+0000", 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4219e-05 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.012353+0000", 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012883860000000001 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.012405+0000", 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2500000000000002e-05 2024-04-18T21:52:28.948 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 62/43 ECSubWrite(tid=36, reqid=osd.6.0:36, at_version=62'30, trim_to=0'0, roll_forward_to=61'28))", 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:18.560916+0000", 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.320959362, 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.210145524, 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.949 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.560916+0000", 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.560916+0000", 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.560916+0000", 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.92e-07 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.560943+0000", 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7387000000000001e-05 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.560952+0000", 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5269999999999998e-06 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.661059+0000", 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10010732899999999 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.761277+0000", 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10021778200000001 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.761294+0000", 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6929999999999999e-05 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.770996+0000", 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0097020209999999999 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.771061+0000", 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5356000000000002e-05 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:474 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~688128] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 9.8767410550000001, 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31877274300000003, 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.953 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 474 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005137+0000", 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000001e-06 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005928+0000", 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00079063400000000002 2024-04-18T21:52:28.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005960+0000", 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2727e-05 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005976+0000", 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5937e-05 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014929+0000", 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0089524860000000008 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.015108+0000", 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000179366 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.017762+0000", 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026539900000000002 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106305+0000", 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.088543338999999999 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.323764+0000", 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.217458182 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.323907+0000", 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014321200000000001 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:475 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~540672] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:28.958 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 9.8677974279999994, 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31080646899999997, 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 475 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014079+0000", 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.72e-06 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014732+0000", 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065293199999999995 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014757+0000", 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4352999999999999e-05 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014770+0000", 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.324e-05 2024-04-18T21:52:28.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.020116+0000", 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053463540000000002 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.020404+0000", 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028811800000000002 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.022644+0000", 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022399899999999999 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106309+0000", 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.083664469000000005 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.324710+0000", 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21840135099999999 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.324884+0000", 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017394200000000001 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:476 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~229376] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 9.8634702179999998, 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30710270099999998, 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.964 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 476 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018408+0000", 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9660000000000001e-06 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018913+0000", 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00050535599999999999 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018956+0000", 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2740000000000001e-05 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019019+0000", 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2694000000000007e-05 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.023815+0000", 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047959430000000004 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.024087+0000", 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027221100000000001 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.025513+0000", 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014256360000000001 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106311+0000", 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.080798549999999997 2024-04-18T21:52:28.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.325343+0000", 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21903152300000001 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.325508+0000", 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016508199999999999 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:477 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 9.8628169910000008, 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40712708600000003, 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 477 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.970 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019060+0000", 2024-04-18T21:52:28.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7859999999999999e-06 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019070+0000", 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0345e-05 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019098+0000", 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7392000000000001e-05 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019115+0000", 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7765e-05 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.026938+0000", 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078220470000000004 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.027160+0000", 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000222625 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:28.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.027978+0000", 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081757100000000001 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106314+0000", 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.078336021000000006 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.426038+0000", 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31972404500000001 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.426185+0000", 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000147489 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:478 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 9.8627320629999993, 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.709574606, 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:28.975 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 478 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019144+0000", 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.206e-06 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019151+0000", 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4819999999999999e-06 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019170+0000", 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8906000000000002e-05 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019185+0000", 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5116999999999999e-05 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.028449+0000", 2024-04-18T21:52:28.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0092640520000000001 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.028689+0000", 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023998100000000001 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.427127+0000", 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39843774999999998 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:28.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.427384+0000", 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025700499999999998 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.728718+0000", 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30133410700000002 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:28.980 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:28.981 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:28.981 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:28.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:28.983 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:28.983 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:28.984 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:28.980+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:29.083 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:29.079+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:29.151 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:29.151 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:29.151 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:29.151 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:29.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:29.184 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:29.180+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:29.283 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:29.283 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:29.283 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:29.283 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:29.283 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:29.284 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:29.283+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:29.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:29.391 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:29.387+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:29.404 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:29.383+0000 7efd122ee640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:29.404 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:29.383+0000 7efd122ee640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:29.404 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:29.387+0000 7efd122ee640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:29.405 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:29.387+0000 7efd122ee640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:29.406 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:29.391+0000 7efd122ee640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:29.406 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:29.391+0000 7efd122ee640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:29.427 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.8s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.661021707000003, 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51395767000000003, 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.428 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764361+0000", 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.61e-06 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764368+0000", 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1600000000000001e-06 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764389+0000", 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1073e-05 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764409+0000", 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0006000000000001e-05 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866089+0000", 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101679558 2024-04-18T21:52:29.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866221+0000", 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132022 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278264+0000", 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41204366100000001 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278317+0000", 2024-04-18T21:52:29.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2580000000000001e-05 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:103 3.18s0 3:1b279e7b:::smithi09731216-35:head [append 0~573440,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.444363242999998, 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6106588159999999, 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.433 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 103 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981021+0000", 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5470000000000001e-06 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981778+0000", 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00075675099999999995 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981811+0000", 2024-04-18T21:52:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3200000000000001e-05 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981832+0000", 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0703000000000002e-05 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.277997+0000", 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29616472700000002 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:29.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278003+0000", 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3269999999999997e-06 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591581+0000", 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.313578111 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591677+0000", 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5450000000000003e-05 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:104 3.18s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.443498388999998, 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714877378, 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.438 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981885+0000", 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0260000000000002e-06 2024-04-18T21:52:29.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981895+0000", 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7969999999999995e-06 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981918+0000", 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3116e-05 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981931+0000", 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2850000000000001e-05 2024-04-18T21:52:29.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278086+0000", 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29615502199999999 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278088+0000", 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6230000000000001e-06 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696611+0000", 2024-04-18T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418522515 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696760+0000", 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149429 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.443434083, 2024-04-18T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9142408849999999, 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981948+0000", 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981952+0000", 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8739999999999997e-06 2024-04-18T21:52:29.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981963+0000", 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.133e-05 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981971+0000", 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2900000000000002e-06 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278161+0000", 2024-04-18T21:52:29.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29618945299999999 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278163+0000", 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.734e-06 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696819+0000", 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418656006 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.697058+0000", 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023910700000000001 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.896188+0000", 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19913024800000001 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:227 3.19s0 3:9941efaf:::smithi09731216-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:29.447 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.412221803000001, 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50662614500000003, 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 227 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013160+0000", 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1299999999999999e-07 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013167+0000", 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9670000000000004e-06 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013187+0000", 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9726000000000001e-05 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013202+0000", 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5483000000000001e-05 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013460+0000", 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025794499999999998 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013468+0000", 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8460000000000008e-06 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.315837+0000", 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30236893999999997 2024-04-18T21:52:29.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.316131+0000", 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029429400000000001 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.519785+0000", 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20365413099999999 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.452 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:318 3.19s0 3:9da3af1e:::smithi09731216-23:head [delete] snapc 5=[2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 31.232508261, 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60663671399999997, 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 318 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192874+0000", 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2699999999999998e-07 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192878+0000", 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0239999999999999e-06 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192886+0000", 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2549999999999997e-06 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192890+0000", 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.357e-06 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372031+0000", 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.179140625 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372092+0000", 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1457999999999993e-05 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372626+0000", 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053349599999999995 2024-04-18T21:52:29.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.374211+0000", 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001584752 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799295+0000", 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42508415500000002 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799510+0000", 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021466500000000001 2024-04-18T21:52:29.457 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:329 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.201732988, 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60620628200000004, 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.458 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 329 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6600000000000003e-07 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223651+0000", 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5919999999999999e-06 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223655+0000", 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9890000000000003e-06 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223658+0000", 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2600000000000001e-06 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514422+0000", 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29076426599999999 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514584+0000", 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000161275 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625434+0000", 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.110850424 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625669+0000", 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234315 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.829854+0000", 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20418589500000001 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:360 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~475136] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.061194837999999, 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98249101000000005, 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 360 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364189+0000", 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8420000000000001e-06 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364486+0000", 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029724500000000002 2024-04-18T21:52:29.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364517+0000", 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0994000000000002e-05 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364530+0000", 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2330999999999999e-05 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364644+0000", 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114673 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364657+0000", 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2258e-05 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.394826+0000", 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.030169702 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395495+0000", 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00066827599999999998 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.821805+0000", 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42631051199999997 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.822025+0000", 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000220084 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824602+0000", 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025771510000000002 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944319+0000", 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11971646900000001 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346144+0000", 2024-04-18T21:52:29.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401824764 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346677+0000", 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053370900000000005 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:361 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~622592 in=622592b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.056454606999999, 2024-04-18T21:52:29.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2802270259999999, 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 361 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:29.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368928+0000", 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6559999999999999e-06 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369217+0000", 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028880699999999997 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369238+0000", 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0914999999999999e-05 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369247+0000", 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1910000000000006e-06 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369333+0000", 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5482000000000006e-05 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369345+0000", 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2147000000000001e-05 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395635+0000", 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026290322000000001 2024-04-18T21:52:29.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395761+0000", 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126348 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837828+0000", 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44206649199999998 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837999+0000", 2024-04-18T21:52:29.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017102900000000001 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.840298+0000", 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022987670000000002 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944332+0000", 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10403438199999999 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:29.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648919+0000", 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70458679599999996 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649154+0000", 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234692 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:362 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~147456 in=147456b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:29.477 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.055474981, 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.279510908, 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 362 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:29.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369907+0000", 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0529999999999999e-06 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369994+0000", 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6688999999999996e-05 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370003+0000", 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3789999999999995e-06 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370010+0000", 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9229999999999996e-06 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370098+0000", 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8146000000000006e-05 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370122+0000", 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3329e-05 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395809+0000", 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025687331000000001 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395945+0000", 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013612499999999999 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941837+0000", 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54589167500000002 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941940+0000", 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103435 2024-04-18T21:52:29.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942633+0000", 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069312599999999996 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244973+0000", 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30233927599999999 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:29.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649234+0000", 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40426141100000001 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649417+0000", 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018301 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:363 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:29.484 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.055358807000001, 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.380372242, 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 363 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370023+0000", 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.75e-07 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370027+0000", 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8489999999999999e-06 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370039+0000", 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2013e-05 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370043+0000", 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6630000000000001e-06 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370169+0000", 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125506 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370180+0000", 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0798000000000001e-05 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396197+0000", 2024-04-18T21:52:29.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026017723999999999 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396320+0000", 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122905 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943149+0000", 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54682909400000002 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943272+0000", 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012215699999999999 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943899+0000", 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000627106 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244975+0000", 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30107650600000002 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.749791+0000", 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50481555700000003 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.750395+0000", 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060378899999999998 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:365 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~491520] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.051749074, 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97351366699999997, 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 365 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373634+0000", 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.024e-06 2024-04-18T21:52:29.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373873+0000", 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000238459 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373899+0000", 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6461999999999998e-05 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373912+0000", 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2823000000000001e-05 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374028+0000", 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115758 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374050+0000", 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2314e-05 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.601683+0000", 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22763349599999999 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.602117+0000", 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043313999999999999 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828190+0000", 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226073089 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828481+0000", 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029162099999999999 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.830297+0000", 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018160229999999999 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944323+0000", 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114025979 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346866+0000", 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40254325000000002 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.347146+0000", 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027922899999999998 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.498 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:366 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~770048] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.046130142999999, 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.067907554, 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 366 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379253+0000", 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9609999999999999e-06 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379714+0000", 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046141699999999999 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379730+0000", 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5585000000000001e-05 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379738+0000", 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379830+0000", 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2551000000000002e-05 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379852+0000", 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.143e-05 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.604888+0000", 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225036559 2024-04-18T21:52:29.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.605008+0000", 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119936 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831601+0000", 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226592508 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831778+0000", 2024-04-18T21:52:29.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017783799999999999 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.834089+0000", 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023105930000000001 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944325+0000", 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11023567500000001 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:29.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.446970+0000", 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50264567000000004 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.447159+0000", 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018811199999999999 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:367 3.11s0 3:8af06b78:::smithi09731216-10:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:29.505 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.045625225999999, 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1678633460000001, 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 367 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4899999999999995e-07 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379760+0000", 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4170000000000001e-06 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379766+0000", 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7459999999999997e-06 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379769+0000", 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.608e-06 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379929+0000", 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015980200000000001 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379953+0000", 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3764000000000002e-05 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707668+0000", 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32771509500000001 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707855+0000", 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186987 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835388+0000", 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127533496 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835835+0000", 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044705299999999999 2024-04-18T21:52:29.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.836767+0000", 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00093163299999999996 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944330+0000", 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107563336 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547378+0000", 2024-04-18T21:52:29.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60304798000000004 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547619+0000", 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024088000000000001 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:368 3.11s0 3:8af06b78:::smithi09731216-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:29.512 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.045589295999999, 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6760458580000002, 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 368 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:29.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.91e-07 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379799+0000", 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2690000000000002e-06 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379805+0000", 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7989999999999999e-06 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379809+0000", 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4929999999999998e-06 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379996+0000", 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186959 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.380015+0000", 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9837e-05 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707952+0000", 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32793653499999997 2024-04-18T21:52:29.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.708113+0000", 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160536 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837592+0000", 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12947945499999999 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837760+0000", 2024-04-18T21:52:29.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016796899999999999 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.647833+0000", 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.81007341499999996 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648038+0000", 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020446900000000001 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:05.055838+0000", 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.407799831 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=50, last_complete=47'14, committed=1, applied=1))", 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 25.197992323000001, 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42177490000000001, 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.519 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:29.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8000000000000002e-07 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227392+0000", 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.57e-06 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227402+0000", 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9650000000000003e-06 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327516+0000", 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100114438 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648849+0000", 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32133336099999998 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649164+0000", 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031428599999999999 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.522 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=50, last_complete=47'14, committed=1, applied=1))", 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 25.190778409, 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41408854499999997, 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5199999999999998e-07 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234605+0000", 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0229999999999999e-06 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234611+0000", 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9499999999999998e-06 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234615+0000", 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2100000000000003e-06 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648672+0000", 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41405668699999998 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648691+0000", 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9423e-05 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=51, last_complete=47'15, committed=1, applied=1))", 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 25.189414893999999, 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41281960600000001, 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235967+0000", 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6899999999999998e-07 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235970+0000", 2024-04-18T21:52:29.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3060000000000002e-06 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235981+0000", 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.083e-05 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235987+0000", 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9309999999999996e-06 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648743+0000", 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41275640400000002 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648786+0000", 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2666000000000001e-05 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=52, last_complete=47'16, committed=1, applied=1))", 2024-04-18T21:52:29.530 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 25.097788284, 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42417700800000002, 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:29.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3099999999999999e-07 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327595+0000", 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8700000000000001e-06 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327600+0000", 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0010000000000001e-06 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327603+0000", 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5429999999999998e-06 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.749596+0000", 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.421992433 2024-04-18T21:52:29.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.751770+0000", 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021739300000000001 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:469 3.11s0 3:8fad7925:::smithi09731216-43:head [read 0~1966080,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.855390550999999, 2024-04-18T21:52:29.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50551296700000004, 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 469 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569992+0000", 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.034e-06 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570000+0000", 2024-04-18T21:52:29.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1680000000000004e-06 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570022+0000", 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2595999999999998e-05 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570033+0000", 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0858e-05 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.571032+0000", 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00099923299999999998 2024-04-18T21:52:29.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.571153+0000", 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120982 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.075504+0000", 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50435009600000003 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.538 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:29.539 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:29.539 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:29.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:29.541 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:29.540+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:29.641 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:29.640+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:29.672 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:29.672 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:29.672 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:29.672 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:29.686 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:29.742 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:29.739+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:29.807 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:29.807 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:29.807 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:29.807 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:29.807 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:29.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:29.869 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:29.867+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:29.943 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:29.939+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.693721195000002, 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82320560300000001, 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.952 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254837+0000", 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9000000000000002e-06 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255465+0000", 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000628435 2024-04-18T21:52:29.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255498+0000", 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2305999999999999e-05 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255523+0000", 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5137e-05 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078012+0000", 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82248975599999996 2024-04-18T21:52:29.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078039+0000", 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7069000000000001e-05 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.692994290000001, 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82259326700000002, 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.956 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255563+0000", 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.72e-06 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255574+0000", 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0993e-05 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255598+0000", 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4329999999999999e-05 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255615+0000", 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7569000000000001e-05 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078131+0000", 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82251584300000002 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078154+0000", 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2812000000000001e-05 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:93 3.3s0 3.fef99513 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.692906348000001, 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822627518, 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.960 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 93 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255650+0000", 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2330000000000001e-06 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255657+0000", 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5000000000000002e-06 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255681+0000", 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3354e-05 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255705+0000", 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3932999999999999e-05 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078240+0000", 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822535609 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078276+0000", 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5889000000000001e-05 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.963 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:73 3.13s0 3:c8da3643:::smithi09731216-25:head [append 0~409600,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 42.751822877999999, 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.609707234, 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.964 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196734+0000", 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829999999999999e-06 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196977+0000", 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024357200000000001 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196998+0000", 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0503e-05 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197013+0000", 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5536000000000002e-05 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197060+0000", 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6519e-05 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197061+0000", 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2900000000000002e-07 2024-04-18T21:52:29.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591261+0000", 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3942002449999999 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591644+0000", 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038315000000000002 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593905+0000", 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022609050000000001 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697736+0000", 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10383110700000001 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806236+0000", 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10850008999999999 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806439+0000", 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020309500000000001 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:74 3.13s0 3:c8da3643:::smithi09731216-25:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 42.751518533999999, 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7102205860000002, 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.970 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 74 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197038+0000", 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9000000000000005e-07 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197043+0000", 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197053+0000", 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0267e-05 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197066+0000", 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.327e-05 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197104+0000", 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8039e-05 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197105+0000", 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8400000000000004e-07 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595282+0000", 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3981771869999999 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595543+0000", 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026070899999999997 2024-04-18T21:52:29.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.596517+0000", 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00097469500000000003 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697956+0000", 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10143811899999999 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907075+0000", 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20911996799999999 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907257+0000", 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018164799999999999 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.13s0 3:c8a99f7f:::smithi09731216-31:head [append 0~507904,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:29.976 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 42.750992471000004, 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.812483013, 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8599999999999998e-07 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197879+0000", 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000315495 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197893+0000", 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4143e-05 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197908+0000", 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4932999999999999e-05 2024-04-18T21:52:29.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197987+0000", 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9288000000000004e-05 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197989+0000", 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.137e-06 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.698727+0000", 2024-04-18T21:52:29.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.500737242 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.699015+0000", 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028849499999999999 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.700958+0000", 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019429969999999999 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.981 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798717+0000", 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097759288999999999 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.009932+0000", 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.211214651 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010046+0000", 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113657 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.982 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.13s0 3:c8a99f7f:::smithi09731216-31:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 42.750630630000003, 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8124102710000001, 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.983 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197925+0000", 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000005e-07 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197931+0000", 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6799999999999998e-06 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197943+0000", 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1620000000000001e-05 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197953+0000", 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-05 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7586000000000001e-05 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:29.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5599999999999995e-07 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702431+0000", 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.504419629 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702670+0000", 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023948700000000001 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.703977+0000", 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00130674 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798719+0000", 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094742397000000006 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010219+0000", 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21149920999999999 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010335+0000", 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011602 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:232 3.3s0 3:c170a397:::smithi09731216-3:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.915473261999999, 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.67408736800000002, 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.989 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 232 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033085+0000", 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.74e-06 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033688+0000", 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060374300000000001 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033720+0000", 2024-04-18T21:52:29.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2119000000000002e-05 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033737+0000", 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6273000000000002e-05 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.299553+0000", 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.26581622199999999 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.299658+0000", 2024-04-18T21:52:29.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105429 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.300593+0000", 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093458899999999999 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.304845+0000", 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042519960000000001 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:29.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707032+0000", 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.402187454 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707169+0000", 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013680300000000001 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:233 3.3s0 3:c170a397:::smithi09731216-3:head [setxattr _header (55) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:52:29.994 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.914785627999997, 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.67365432700000005, 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 233 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:29.995 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033771+0000", 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2500000000000001e-06 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033776+0000", 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7629999999999997e-06 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033791+0000", 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4803e-05 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033801+0000", 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6830000000000003e-06 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.302979+0000", 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.26917756799999998 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.303256+0000", 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027797400000000001 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.304206+0000", 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094934700000000002 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.505420+0000", 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201213902 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707355+0000", 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20193533899999999 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707424+0000", 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8697999999999999e-05 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:29.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:234 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.914738501999999, 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.87684132699999995, 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 234 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.000 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9599999999999998e-07 2024-04-18T21:52:30.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033821+0000", 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.653e-06 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033834+0000", 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3303e-05 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033847+0000", 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2327000000000001e-05 2024-04-18T21:52:30.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808105+0000", 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.77425798199999996 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808280+0000", 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017508500000000001 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.910658+0000", 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102378081 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.003 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:306 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.862222228, 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71140329400000002, 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 306 2024-04-18T21:52:30.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086334+0000", 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4900000000000004e-07 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086340+0000", 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0900000000000001e-06 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086359+0000", 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8940000000000002e-05 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086369+0000", 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7569999999999999e-06 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086608+0000", 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000239702 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086634+0000", 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5948999999999998e-05 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.291884+0000", 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20524995300000001 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292051+0000", 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000166797 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.797736+0000", 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50568515700000005 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.008 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:307 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.862172139999998, 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81219510800000005, 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 307 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.009 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086384+0000", 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9699999999999995e-07 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086388+0000", 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3900000000000003e-06 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086398+0000", 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8369999999999992e-06 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086406+0000", 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3599999999999996e-06 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086727+0000", 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032101900000000002 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086748+0000", 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0917e-05 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292674+0000", 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20592602600000001 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292826+0000", 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015169700000000001 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.898578+0000", 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60575206500000001 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.013 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:325 3.3s0 3:c26b2dda:::smithi09731216-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 30.722381253999998, 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.79501891000000002, 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.014 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 325 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4400000000000001e-07 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226176+0000", 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1100000000000001e-06 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226182+0000", 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5230000000000003e-06 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226186+0000", 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0180000000000003e-06 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.230252+0000", 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040660369999999998 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.230395+0000", 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014306400000000001 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.817971+0000", 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58757587300000003 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.818220+0000", 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024919800000000001 2024-04-18T21:52:30.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:00.021193+0000", 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20297274300000001 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:425 3.3s0 3:c022028e:::smithi09731216-15:head [copy-from ver 14] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445446+0000", 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.503108607, 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.72780147399999995, 2024-04-18T21:52:30.019 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 425 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445446+0000", 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445446+0000", 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445450+0000", 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7129999999999999e-06 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445466+0000", 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.543e-05 2024-04-18T21:52:30.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445506+0000", 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0049000000000002e-05 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445531+0000", 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.552e-05 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445699+0000", 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016749399999999999 2024-04-18T21:52:30.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445711+0000", 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2616000000000001e-05 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.452009+0000", 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0062980550000000003 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.452196+0000", 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018673999999999999 2024-04-18T21:52:30.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.867038+0000", 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41484197699999997 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.873483+0000", 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0064446649999999996 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173090+0000", 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299607611 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173248+0000", 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015760399999999999 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:432 3.3s0 3:c170a397:::smithi09731216-3:head [copy-from ver 11] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.502843401, 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.728029657, 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.025 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 432 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445713+0000", 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4759999999999999e-06 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445723+0000", 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.027e-05 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445745+0000", 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.126e-05 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445770+0000", 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5803e-05 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445905+0000", 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013424800000000001 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445914+0000", 2024-04-18T21:52:30.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9230000000000008e-06 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.552790+0000", 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106876007 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.553298+0000", 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00050816200000000005 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.967658+0000", 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41436041099999998 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.072477+0000", 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104818595 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173605+0000", 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101128601 2024-04-18T21:52:30.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173741+0000", 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013590100000000001 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:529 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~786432] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.405381+0000", 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.5431744890000001, 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62624382199999995, 2024-04-18T21:52:30.031 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 529 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.405381+0000", 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.405381+0000", 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.405383+0000", 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5129999999999999e-06 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.406636+0000", 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001253288 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.406667+0000", 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1022999999999998e-05 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.406683+0000", 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5985999999999999e-05 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409012+0000", 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002328161 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409201+0000", 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000189827 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.411632+0000", 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024307999999999999 2024-04-18T21:52:30.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.519543+0000", 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107911147 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.031484+0000", 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51194032899999997 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.031624+0000", 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000140748 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:530 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~491520] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.408107+0000", 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.5404475810000005, 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62388588300000003, 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.037 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 530 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408107+0000", 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408107+0000", 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408110+0000", 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6000000000000001e-06 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408800+0000", 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069015799999999996 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408832+0000", 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1807000000000002e-05 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408849+0000", 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6786000000000002e-05 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.513965+0000", 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105116449 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.514105+0000", 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000139877 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.515322+0000", 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012172109999999999 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.520716+0000", 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053939879999999997 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.031865+0000", 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51114865499999995 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.031993+0000", 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012835200000000001 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.042 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:531 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~278528] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.409700+0000", 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.538854937, 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62260299100000005, 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 531 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409700+0000", 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409700+0000", 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409702+0000", 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1059999999999998e-06 2024-04-18T21:52:30.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410069+0000", 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036714499999999998 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410099+0000", 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9577000000000002e-05 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410117+0000", 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8012999999999998e-05 2024-04-18T21:52:30.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.516213+0000", 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106095574 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.516497+0000", 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028478299999999999 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.518393+0000", 2024-04-18T21:52:30.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018953959999999999 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.521659+0000", 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032665699999999999 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.032199+0000", 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51053993200000003 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.032303+0000", 2024-04-18T21:52:30.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010389499999999999 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:532 3.3s0 3:c2fd059f:::smithi09731216-20:head [setxattr _header (57) in=7b] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.410147+0000", 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.5384081460000001, 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.72290332300000004, 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.048 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 532 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410147+0000", 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410147+0000", 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410148+0000", 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.083e-06 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410153+0000", 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2460000000000003e-06 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410167+0000", 2024-04-18T21:52:30.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3733999999999999e-05 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510246+0000", 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078801 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.925981+0000", 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41573498199999998 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.926145+0000", 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016407100000000001 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.926979+0000", 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00083374299999999996 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.928947+0000", 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001968479 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.132932+0000", 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.203985165 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133050+0000", 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118019 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:533 3.3s0 3:c2fd059f:::smithi09731216-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:30.053 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.4382897010000004, 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.85709232000000002, 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 533 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510266+0000", 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-06 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510271+0000", 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7670000000000003e-06 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510283+0000", 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1965e-05 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.610415+0000", 2024-04-18T21:52:30.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100132151 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.927765+0000", 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31734968699999999 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.927918+0000", 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000152702 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133621+0000", 2024-04-18T21:52:30.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.205703102 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133704+0000", 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3040000000000005e-05 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.367358+0000", 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23365385799999999 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.058 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.059 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.059 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.059 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:30.060 INFO:tasks.rados.rados.0.smithi097.stdout:226: done (1 left) 2024-04-18T21:52:30.060 INFO:tasks.rados.rados.0.smithi097.stdout:227: copy_from oid 41 from oid 11 current snap is 19 2024-04-18T21:52:30.060 INFO:tasks.rados.rados.0.smithi097.stdout:228: delete oid 28 current snap is 19 2024-04-18T21:52:30.061 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:30.061 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:30.055+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:30.161 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:30.155+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:30.227 INFO:tasks.rados.rados.0.smithi097.stdout:227: finishing copy_from racing read to smithi09731216-41 2024-04-18T21:52:30.239 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:30.239 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:30.239 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:30.239 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:30.253 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:30.297 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:30.295+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:30.362 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:30.360+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:30.373 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:30.373 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:30.373 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:30.373 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:30.374 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:30.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:30.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-18T21:52:30.409 INFO:tasks.rados.rados.0.smithi097.stdout:228: done (2 left) 2024-04-18T21:52:30.409 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 48 current snap is 19 2024-04-18T21:52:30.409 INFO:tasks.rados.rados.0.smithi097.stdout:229: seq_num 98 ranges {65536=491520,557056=589824,1146880=557056,1703936=16384} 2024-04-18T21:52:30.420 INFO:tasks.rados.rados.0.smithi097.stdout:229: writing smithi09731216-48 from 65536 to 557056 tid 1 2024-04-18T21:52:30.430 INFO:tasks.rados.rados.0.smithi097.stdout:229: writing smithi09731216-48 from 557056 to 1146880 tid 2 2024-04-18T21:52:30.441 INFO:tasks.rados.rados.0.smithi097.stdout:229: writing smithi09731216-48 from 1146880 to 1703936 tid 3 2024-04-18T21:52:30.441 INFO:tasks.rados.rados.0.smithi097.stdout:229: writing smithi09731216-48 from 1703936 to 1720320 tid 4 2024-04-18T21:52:30.442 INFO:tasks.rados.rados.0.smithi097.stdout:230: read oid 25 snap -1 2024-04-18T21:52:30.442 INFO:tasks.rados.rados.0.smithi097.stdout:230: expect deleted 2024-04-18T21:52:30.442 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 2 current snap is 19 2024-04-18T21:52:30.442 INFO:tasks.rados.rados.0.smithi097.stdout:227: finishing copy_from to smithi09731216-41 2024-04-18T21:52:30.442 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 41 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-18T21:52:30.442 INFO:tasks.rados.rados.0.smithi097.stdout:231: seq_num 99 ranges {32768=507904,540672=606208,1146880=557056} 2024-04-18T21:52:30.450 INFO:tasks.rados.rados.0.smithi097.stdout:231: writing smithi09731216-2 from 32768 to 540672 tid 1 2024-04-18T21:52:30.460 INFO:tasks.rados.rados.0.smithi097.stdout:231: writing smithi09731216-2 from 540672 to 1146880 tid 2 2024-04-18T21:52:30.464 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:30.460+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:30.469 INFO:tasks.rados.rados.0.smithi097.stdout:231: writing smithi09731216-2 from 1146880 to 1703936 tid 3 2024-04-18T21:52:30.469 INFO:tasks.rados.rados.0.smithi097.stdout:227: done (4 left) 2024-04-18T21:52:30.469 INFO:tasks.rados.rados.0.smithi097.stdout:232: delete oid 13 current snap is 19 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:100 3.cs0 3:348e44d8:::smithi09731216-34:head [append 0~557056,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.896161204000002, 2024-04-18T21:52:30.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50652407300000002, 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:30.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659953+0000", 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.821e-06 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660381+0000", 2024-04-18T21:52:30.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042725800000000002 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660395+0000", 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.394e-05 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660404+0000", 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6520000000000001e-06 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760564+0000", 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10015995900000001 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760699+0000", 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013526 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762105+0000", 2024-04-18T21:52:30.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00140595 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965039+0000", 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20293309400000001 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166421+0000", 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201382373 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166476+0000", 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4765999999999997e-05 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:101 3.cs0 3:348e44d8:::smithi09731216-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.895691464000002, 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60956795200000002, 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.565 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660422+0000", 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-07 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660426+0000", 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3830000000000002e-06 2024-04-18T21:52:30.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660436+0000", 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8679999999999994e-06 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660443+0000", 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2999999999999998e-06 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762872+0000", 2024-04-18T21:52:30.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1024294 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762973+0000", 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001015 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763506+0000", 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053209199999999996 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965047+0000", 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20154100999999999 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269839+0000", 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.304792377 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269989+0000", 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015033399999999999 2024-04-18T21:52:30.570 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:102 3.cs0 3:348e44d8:::smithi09731216-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.895641914999999, 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.92759091000000005, 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.571 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660472+0000", 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660476+0000", 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0879999999999997e-06 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660493+0000", 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6963e-05 2024-04-18T21:52:30.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660502+0000", 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9579999999999996e-06 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763793+0000", 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10329121600000001 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:30.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763891+0000", 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8127000000000004e-05 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372039+0000", 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60814838199999999 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372189+0000", 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014991200000000001 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.588062+0000", 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.215872654 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.5s0 30/26 ECSubReadReply(tid=19, attrs_read=0))", 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.796247426999997, 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30082227499999997, 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759865+0000", 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759866+0000", 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.34e-07 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759873+0000", 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.973e-06 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759879+0000", 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5440000000000004e-06 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.759885+0000", 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3340000000000001e-06 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.060439+0000", 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30055447000000002 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.060688+0000", 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024862000000000002 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.579 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.790480885999997, 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299971873, 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765632+0000", 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1800000000000001e-07 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765635+0000", 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6450000000000001e-06 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.765644+0000", 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.755e-06 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.865734+0000", 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10008987699999999 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065540+0000", 2024-04-18T21:52:30.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19980606200000001 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.065604+0000", 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4115999999999994e-05 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 30/26 ECSubWriteReply(tid=21, last_complete=30'8, committed=1, applied=1))", 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.689418130999996, 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299440857, 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.583 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866695+0000", 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.13e-07 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866703+0000", 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.866722+0000", 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9522999999999999e-05 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.966809+0000", 2024-04-18T21:52:30.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10008695400000001 2024-04-18T21:52:30.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166119+0000", 2024-04-18T21:52:30.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19931021099999999 2024-04-18T21:52:30.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166136+0000", 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6319000000000001e-05 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:78 3.15s0 3:ab33d1b5:::smithi09731216-26:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:30.587 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.575096490999996, 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1076682130000002, 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981019+0000", 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.447e-06 2024-04-18T21:52:30.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981025+0000", 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6810000000000001e-06 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981044+0000", 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8856999999999999e-05 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981056+0000", 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1921e-05 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164029+0000", 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18297308600000001 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164033+0000", 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0369999999999996e-06 2024-04-18T21:52:30.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885490+0000", 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7214569900000001 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885771+0000", 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028064300000000002 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.088685+0000", 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.202913551 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:133 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.545830494, 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.393294378, 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.593 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 133 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010282+0000", 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010286+0000", 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012871+0000", 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025854699999999999 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012896+0000", 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5409e-05 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012911+0000", 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.484e-05 2024-04-18T21:52:30.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013086+0000", 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017422499999999999 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013437+0000", 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035166100000000002 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.016765+0000", 2024-04-18T21:52:30.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033273949999999999 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.020122+0000", 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033576370000000001 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403436+0000", 2024-04-18T21:52:30.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38331330800000002 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403577+0000", 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014114999999999999 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:134 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.543128967999998, 2024-04-18T21:52:30.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39119692299999997, 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 134 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012984+0000", 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012986+0000", 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0839999999999999e-06 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.012995+0000", 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4159999999999993e-06 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013025+0000", 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0035999999999998e-05 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.013046+0000", 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0720000000000002e-05 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.017993+0000", 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049471020000000001 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.018258+0000", 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026496000000000001 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.019560+0000", 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001301373 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.020585+0000", 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010257910000000001 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.403896+0000", 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38331068899999998 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.404181+0000", 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028475199999999999 2024-04-18T21:52:30.604 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:237 3.cs0 3:3bc99ad3:::smithi09731216-27:head [read 0~442368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.522868398, 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30619164900000001, 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.605 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033244+0000", 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033245+0000", 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3499999999999996e-07 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033248+0000", 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3730000000000001e-06 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033256+0000", 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8509999999999999e-06 2024-04-18T21:52:30.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033262+0000", 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4480000000000002e-06 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033435+0000", 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000173847 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:30.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033439+0000", 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6540000000000001e-06 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.237425+0000", 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20398589 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.237506+0000", 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1211000000000006e-05 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.339436+0000", 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10192983999999999 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 40/26 ECSubWriteReply(tid=53, last_complete=40'21, committed=1, applied=1))", 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.034487931999998, 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.409140051, 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.610 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0900000000000002e-07 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521629+0000", 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0169999999999998e-06 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521640+0000", 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0246e-05 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521647+0000", 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9070000000000001e-06 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.929171+0000", 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40752403300000001 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.930765+0000", 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001594339 2024-04-18T21:52:30.613 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:463 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~638976] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.182023642000001, 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50349991299999997, 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.614 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 463 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374091+0000", 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.84e-06 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375137+0000", 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010459 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375156+0000", 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978e-05 2024-04-18T21:52:30.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375169+0000", 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2660000000000001e-05 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571128+0000", 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19595929000000001 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571311+0000", 2024-04-18T21:52:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018294599999999999 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572699+0000", 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001388715 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675602+0000", 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102902721 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877476+0000", 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20187386299999999 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877589+0000", 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.619 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:464 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~131072] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.180910057, 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50427034999999998, 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 464 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375204+0000", 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3650000000000001e-06 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375344+0000", 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013967800000000001 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375357+0000", 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3580999999999999e-05 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375367+0000", 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0117e-05 2024-04-18T21:52:30.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573497+0000", 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198129889 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573606+0000", 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010875 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574480+0000", 2024-04-18T21:52:30.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00087374900000000005 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675605+0000", 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10112497400000001 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879333+0000", 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20372839500000001 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879473+0000", 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013985199999999999 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:465 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.180727522, 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60269858399999998, 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.625 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 465 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:30.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375386+0000", 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6300000000000004e-07 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375391+0000", 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.729e-06 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375401+0000", 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9569999999999997e-06 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375409+0000", 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6620000000000005e-06 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777143+0000", 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40173476200000002 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777239+0000", 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5712999999999997e-05 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777695+0000", 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045595 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.778376+0000", 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068123800000000003 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.977952+0000", 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.199575432 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.978084+0000", 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000132278 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:466 3.cs0 3:3c73a4f8:::smithi09731216-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.180692835, 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006726617, 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 466 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375421+0000", 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.009e-06 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375424+0000", 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.495e-06 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375435+0000", 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0360000000000001e-05 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375442+0000", 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6389999999999992e-06 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777929+0000", 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40248657999999998 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777993+0000", 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078687+0000", 2024-04-18T21:52:30.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30069407399999998 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078849+0000", 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162065 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.382147+0000", 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30329795300000001 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 61/26 ECSubWriteReply(tid=88, last_complete=62'48, committed=1, applied=1))", 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.379271+0000", 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.176841909, 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29601272699999998, 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379271+0000", 2024-04-18T21:52:30.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379271+0000", 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379272+0000", 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0559999999999999e-06 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379277+0000", 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7010000000000004e-06 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379289+0000", 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2799999999999999e-05 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.379297+0000", 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5039999999999996e-06 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675220+0000", 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29592301700000001 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675284+0000", 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3649e-05 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.639 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 62/26 ECSubWriteReply(tid=88, last_complete=62'48, committed=1, applied=1))", 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.380752+0000", 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.175360601, 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29469121700000001, 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380752+0000", 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380752+0000", 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380753+0000", 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3600000000000003e-07 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380757+0000", 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.126e-06 2024-04-18T21:52:30.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380769+0000", 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2135e-05 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.380778+0000", 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7190000000000007e-06 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675386+0000", 2024-04-18T21:52:30.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29460853199999998 2024-04-18T21:52:30.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675443+0000", 2024-04-18T21:52:30.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6969000000000003e-05 2024-04-18T21:52:30.645 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.645 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.11s5 62/49 ECSubRead(tid=94, to_read={3:8fad7925:::smithi09731216-43:head=0,491520,0}, subchunks={3:8fad7925:::smithi09731216-43:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.571167+0000", 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.984945636999999, 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.300446565, 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571167+0000", 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571167+0000", 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571168+0000", 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1800000000000005e-07 2024-04-18T21:52:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571178+0000", 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0111e-05 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571190+0000", 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2016e-05 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.671270+0000", 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10007946099999999 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.871478+0000", 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200207947 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.871614+0000", 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000136212 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.649 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:491 3.5s0 3:a2c4e662:::smithi09731216-50:head [create,setxattr `k]abUjc_kPO]\\rQ[m\\eRa\\f]aW]pgQYXV]gVkgc (1435) in=40b,setxattr qen`h]Vg]rpmXc_]tPY^Xpfd (1823) in=24b,setxattr [idOh[aNZjbdcYS\\eZaVVetqi[ric_V (1823) in=31b,setxattr daSce\\mdeZpfaaqjVU^^]PU\\S`[RbRZmXqSkgZQj (1087) in=40b,setxattr YgloTThONOYj[cntZofPOaP^O_bsWg]fd (1823) in=33b,setxattr ejbfqsafT_htneSabdatV`Qp^RS`f[nN``fN\\ebj (807) in=40b,setxattr Ya\\[Sha^pYao_k[dboRNsfOYcXoQtQRfV[XWcTQj (947) in=40b,setxattr YNVc]ZTPiOoc`NXSYsd[NZljlaqNVlYlRnihjYdQ (1789) in=40b,setxattr _t]sstiO]naqp_Zar\\Vtqss]cmN]kYnT_NtflgVd (9120) in=40b,setxattr lXVOWmpdrdYlUkq]]nPfdVVotggPsOZ\\VmPemila (909) in=40b,setxattr slQjq^hbmpO^nVhe\\te\\SkYldVebmeasZVgapUqn (454) in=40b,setxattr n[ea[_dQdc][]\\bfthkjd\\aSbNS^Qe_ZX\\Wr_^^V (594) in=40b,setxattr nS]tbdRhgf]l`\\Tfrnkihlk`Zpgbn[PY`anPZnhr (1682) in=40b,setxattr rpSrTfUUP^cVp_\\d]tNQnmXf[Np\\eO_YeldoslW_ (946) in=40b,setxattr VePmePiYj[hSVdatcprSk\\m]]tXhjbke]jZZYRQS (1681) in=40b,setxattr or[\\eoSZ^ZtXlcRqVTioQ[kZUnqnW[\\ (1823) in=31b,setxattr SS\\`X_`sgfdYjRdpNccP`gcPcRp^eSYXXOOs\\dN[ (1821) in=40b,setxattr obshrdURd^[OaVZZiSTRdPe^cqqgmSO[^Spi^Sgh (1821) in=40b,setxattr oUcnoPgXaRjdRatSbsPd[Ts]f\\pgbOl_Qk^R^Nfq (12280) in=40b,setxattr Xsgptojf]Obssrjj[_NfoopYpXifZhOd\\_]gWoXe (5052) in=40b] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.495516+0000", 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.060596690000001, 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30682174400000001, 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.650 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 491 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495516+0000", 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495516+0000", 2024-04-18T21:52:30.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495520+0000", 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6600000000000001e-06 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495676+0000", 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015610099999999999 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495713+0000", 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7406000000000001e-05 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495736+0000", 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.249e-05 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495905+0000", 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000169204 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.496234+0000", 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032931200000000001 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.497997+0000", 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001762493 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.500587+0000", 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025899999999999999 2024-04-18T21:52:30.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.802032+0000", 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30144495900000001 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.802338+0000", 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030611899999999999 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.16s7 68/31 ECSubWrite(tid=42, reqid=client.4509.0:524, at_version=68'5, trim_to=0'0, roll_forward_to=32'3))", 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.389165+0000", 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1669481279999996, 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30403699299999998, 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.389165+0000", 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.389165+0000", 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.389165+0000", 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3300000000000001e-07 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.390281+0000", 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011157610000000001 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.390322+0000", 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1433000000000001e-05 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.490422+0000", 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100100234 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.490523+0000", 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010018699999999999 2024-04-18T21:52:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.490535+0000", 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2037e-05 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.693160+0000", 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.202625217 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.693202+0000", 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1891000000000003e-05 2024-04-18T21:52:30.659 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.660 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.660 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.660 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.660 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.660 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:30.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:30.662 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:30.659+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:230: done (4 left) 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:232: done (3 left) 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:233: read oid 4 snap -1 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:233: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:234: copy_from oid 26 from oid 29 current snap is 19 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:235: copy_from oid 15 from oid 20 current snap is 19 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:236: read oid 32 snap 18 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:236: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:52:30.742 INFO:tasks.rados.rados.0.smithi097.stdout:237: delete oid 47 current snap is 19 2024-04-18T21:52:30.762 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:30.759+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:30.840 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:30.840 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:574 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~557056 in=557056b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.30397748800000002, 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30402466299999997, 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 574 2024-04-18T21:52:30.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535460+0000", 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.796e-06 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540737+0000", 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052778449999999998 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540774+0000", 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6372999999999999e-05 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540799+0000", 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5743999999999999e-05 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744797+0000", 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20399719899999999 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744963+0000", 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016631799999999999 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.746246+0000", 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012830319999999999 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747841+0000", 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001595352 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.845 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:575 3.as0 3:561bcae7:::smithi09731216-2:head [setxattr _header (56) in=63b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.29859703500000001, 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29878533899999998, 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 575 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540841+0000", 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4540000000000001e-06 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540850+0000", 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2329999999999995e-06 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540874+0000", 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3980000000000001e-05 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540893+0000", 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9173000000000001e-05 2024-04-18T21:52:30.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748051+0000", 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20715831100000001 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748150+0000", 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9155999999999997e-05 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748574+0000", 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042389100000000002 2024-04-18T21:52:30.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.753028+0000", 2024-04-18T21:52:30.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044531889999999998 2024-04-18T21:52:30.850 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.850 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.850 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.853 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:576 3.as0 3:561bcae7:::smithi09731216-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:30.853 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.540918+0000", 2024-04-18T21:52:30.853 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.29851700199999998, 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29882982800000002, 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for rw locks", 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 576 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540918+0000", 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540918+0000", 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540920+0000", 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3599999999999999e-06 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540926+0000", 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0059999999999998e-06 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540949+0000", 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3087000000000001e-05 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540970+0000", 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1126999999999999e-05 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.750911+0000", 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20994078399999999 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.750990+0000", 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8956999999999996e-05 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:588 3.10s0 3:08dd949e:::smithi09731216-47:head [stat,delete] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.836108+0000", 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0033276209999999998, 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037500390000000002, 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for rw locks", 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 588 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.836108+0000", 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.836108+0000", 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.836110+0000", 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6400000000000001e-06 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.836122+0000", 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1996e-05 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.836153+0000", 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1075999999999999e-05 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.836174+0000", 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0287000000000001e-05 2024-04-18T21:52:30.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.837468+0000", 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012940549999999999 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.837820+0000", 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035273200000000002 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.861 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 74/26 ECSubWriteReply(tid=63, last_complete=74'19, committed=1, applied=1))", 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.839847+0000", 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4294967295.9995885, 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017114100000000001, 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839847+0000", 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839847+0000", 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839848+0000", 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0299999999999998e-07 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839853+0000", 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1370000000000001e-06 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839868+0000", 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5636000000000001e-05 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839880+0000", 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1440000000000001e-05 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.864 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGRecoveryDeleteReply(3.10s0 e74,72 3:0e754656:::smithi09731216-19:4,51'19)", 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.839973+0000", 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4294967295.9994617, 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018890399999999999, 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839973+0000", 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839973+0000", 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839975+0000", 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1229999999999998e-06 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.839988+0000", 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2863e-05 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.840022+0000", 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3226999999999999e-05 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.840072+0000", 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0018999999999997e-05 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.867 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:572 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~507904 in=507904b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.30621373600000001, 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30709783899999998, 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 572 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533224+0000", 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.824e-06 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534169+0000", 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094509200000000002 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534205+0000", 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5482999999999999e-05 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534228+0000", 2024-04-18T21:52:30.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2806000000000002e-05 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634530+0000", 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10030222799999999 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634798+0000", 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026831400000000002 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.638371+0000", 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035731249999999999 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744705+0000", 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106333831 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:573 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~606208 in=606208b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:30.872 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.305172428, 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30624561500000003, 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 573 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:30.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534265+0000", 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9800000000000001e-06 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:30.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535350+0000", 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010856869999999999 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535392+0000", 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1192999999999997e-05 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535427+0000", 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5559999999999998e-05 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642525+0000", 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107097461 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642617+0000", 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2181000000000004e-05 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.643719+0000", 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011017449999999999 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:30.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747184+0000", 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103465025 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 8 2024-04-18T21:52:30.877 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:30.878 INFO:tasks.rados.rados.0.smithi097.stdout:234: finishing copy_from to smithi09731216-26 2024-04-18T21:52:30.878 INFO:tasks.rados.rados.0.smithi097.stdout:234: got expected ENOENT (src dne) 2024-04-18T21:52:30.879 INFO:tasks.rados.rados.0.smithi097.stdout:234: finishing copy_from racing read to smithi09731216-26 2024-04-18T21:52:30.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:30.880 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:30.876+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:30.979 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:30.976+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:30.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:31.006 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:31.006 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:31.006 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:31.006 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:31.006 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:31.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:31.042 INFO:tasks.rados.rados.0.smithi097.stdout:229: finishing write tid 1 to smithi09731216-48 2024-04-18T21:52:31.042 INFO:tasks.rados.rados.0.smithi097.stdout:229: finishing write tid 2 to smithi09731216-48 2024-04-18T21:52:31.042 INFO:tasks.rados.rados.0.smithi097.stdout:229: finishing write tid 3 to smithi09731216-48 2024-04-18T21:52:31.047 INFO:tasks.rados.rados.0.smithi097.stdout:229: finishing write tid 4 to smithi09731216-48 2024-04-18T21:52:31.080 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:31.075+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:31.136 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:31.137 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:31.142 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:31.142 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:31.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:31.142 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:31.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.142 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:239 3.as0 3:5bb4af71:::smithi09731216-7:head [delete] snapc 4=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.125591524999997, 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40857630099999998, 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014034+0000", 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2500000000000001e-06 2024-04-18T21:52:31.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014044+0000", 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1889999999999996e-06 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014067+0000", 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3251000000000002e-05 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114145+0000", 2024-04-18T21:52:31.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078193 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114306+0000", 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161044 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114320+0000", 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4044e-05 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317101+0000", 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20278075300000001 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317362+0000", 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026128799999999998 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.319467+0000", 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021048719999999998 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.326280+0000", 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006813375 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422400+0000", 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096119880000000005 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422608+0000", 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020816199999999999 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:308 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~737280] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.049151449, 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69365002799999997, 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 308 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090475+0000", 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.976e-06 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091768+0000", 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00129317 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091803+0000", 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4502999999999998e-05 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091826+0000", 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3235999999999999e-05 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270501+0000", 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.178674888 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270680+0000", 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017917599999999999 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.273556+0000", 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028757610000000001 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281580+0000", 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080240799999999994 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783886+0000", 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50230573000000001 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784122+0000", 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023650799999999999 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:309 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.047070456, 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69216249600000002, 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 309 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.154 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092555+0000", 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-04-18T21:52:31.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093122+0000", 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056744499999999997 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093146+0000", 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128999999999999e-05 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093165+0000", 2024-04-18T21:52:31.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8300000000000001e-05 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275352+0000", 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18218714899999999 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275632+0000", 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028059100000000001 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.277549+0000", 2024-04-18T21:52:31.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001917087 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281584+0000", 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004034513 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784502+0000", 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50291762600000001 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784716+0000", 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021415599999999999 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:310 3.as0 3:5bb4af71:::smithi09731216-7:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.046429285999999, 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69187687499999995, 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 310 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093196+0000", 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093210+0000", 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3517e-05 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093227+0000", 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7050000000000001e-05 2024-04-18T21:52:31.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093243+0000", 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5965999999999999e-05 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279008+0000", 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.185765398 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279287+0000", 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027941400000000002 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.280700+0000", 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412581 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482172+0000", 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20147153900000001 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784956+0000", 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30278430299999998 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785071+0000", 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115454 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:311 3.as0 3:5bb4af71:::smithi09731216-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.046354334, 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99553992999999996, 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 311 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093271+0000", 2024-04-18T21:52:31.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093278+0000", 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1459999999999999e-06 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093295+0000", 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7408999999999999e-05 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093328+0000", 2024-04-18T21:52:31.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2276000000000001e-05 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785901+0000", 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.692573628 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.786181+0000", 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027934999999999999 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.088809+0000", 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30262875700000003 2024-04-18T21:52:31.168 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:427 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.694186608999999, 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38249087900000001, 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.169 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 427 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445437+0000", 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445440+0000", 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2930000000000001e-06 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445454+0000", 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3883000000000001e-05 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445492+0000", 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8090000000000003e-05 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445519+0000", 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6452999999999999e-05 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445801+0000", 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028185900000000003 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445822+0000", 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1418999999999999e-05 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.550566+0000", 2024-04-18T21:52:31.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104743788 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.550788+0000", 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022258600000000001 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.552194+0000", 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001405688 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659810+0000", 2024-04-18T21:52:31.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107616082 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.827786+0000", 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16797619999999999 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.827928+0000", 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000141538 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:428 3.10s0 3:0e754656:::smithi09731216-19:head [rollback 9] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.694049594999999, 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38329886600000002, 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 428 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445574+0000", 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445576+0000", 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.514e-06 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445583+0000", 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6429999999999995e-06 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445626+0000", 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2778999999999998e-05 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445644+0000", 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7597000000000001e-05 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445947+0000", 2024-04-18T21:52:31.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000303462 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445957+0000", 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6059999999999992e-06 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554327+0000", 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10837039 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554582+0000", 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000254906 2024-04-18T21:52:31.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.556153+0000", 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015706660000000001 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659815+0000", 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10366275 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.828747+0000", 2024-04-18T21:52:31.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16893206199999999 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.828873+0000", 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012549100000000001 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:429 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.693947899000001, 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51405278700000001, 2024-04-18T21:52:31.180 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 429 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445677+0000", 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5260000000000001e-06 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445685+0000", 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:52:31.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445703+0000", 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8295999999999999e-05 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445718+0000", 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5383000000000002e-05 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545773+0000", 2024-04-18T21:52:31.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100054752 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545785+0000", 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1783000000000001e-05 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557166+0000", 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011381578 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557443+0000", 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027702800000000002 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.558675+0000", 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012320479999999999 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659884+0000", 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101208162 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959511+0000", 2024-04-18T21:52:31.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29962765699999999 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959729+0000", 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021733700000000001 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.4.19:7 3.as0 3:5bb4af71:::smithi09731216-7:head [assert-version v11,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.584910986000001, 2024-04-18T21:52:31.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40581794100000002, 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554713+0000", 2024-04-18T21:52:31.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554716+0000", 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999999e-06 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554731+0000", 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4722999999999999e-05 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554766+0000", 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5216000000000003e-05 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.554789+0000", 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2495999999999999e-05 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.749822+0000", 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19503310099999999 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.749982+0000", 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016067800000000001 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.960531+0000", 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.210548227 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:495 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~737280] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:31.190 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.632584466000001, 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.664354899, 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 495 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507043+0000", 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.596e-06 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508389+0000", 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013461460000000001 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508432+0000", 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3294999999999998e-05 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508456+0000", 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4134e-05 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581521+0000", 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.073064403 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581827+0000", 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030644999999999999 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.586475+0000", 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046472989999999997 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.699953+0000", 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.113478523 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171243+0000", 2024-04-18T21:52:31.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47128973499999999 2024-04-18T21:52:31.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.196 INFO:tasks.rados.rados.0.smithi097.stdout:229: finishing write tid 5 to smithi09731216-48 2024-04-18T21:52:31.196 INFO:tasks.rados.rados.0.smithi097.stdout:231: finishing write tid 1 to smithi09731216-2 2024-04-18T21:52:31.196 INFO:tasks.rados.rados.0.smithi097.stdout:233: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-18T21:52:31.197 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:31.197 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:31.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171394+0000", 2024-04-18T21:52:31.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000151318 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:496 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~573440] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.629396813, 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76183363900000001, 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.198 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 496 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510229+0000", 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.007e-06 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511015+0000", 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000785792 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511046+0000", 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0873000000000001e-05 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511061+0000", 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5293000000000001e-05 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.691993+0000", 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.180931701 2024-04-18T21:52:31.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.692191+0000", 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019870099999999999 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.694402+0000", 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022106690000000002 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.701975+0000", 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075735730000000001 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.271924+0000", 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56994823500000003 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272061+0000", 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013679499999999999 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:497 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~196608] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.628347451, 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76114050600000005, 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.204 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 497 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511278+0000", 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.387e-06 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511608+0000", 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033003500000000001 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511637+0000", 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9706e-05 2024-04-18T21:52:31.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511661+0000", 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3439999999999999e-05 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696155+0000", 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18449448800000001 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696457+0000", 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000301496 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.698580+0000", 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021231879999999998 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.702633+0000", 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040534079999999997 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272305+0000", 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56967170199999995 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272417+0000", 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111656 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 64/31 ECSubWrite(tid=40, reqid=osd.11.0:40, at_version=64'8, trim_to=0'0, roll_forward_to=62'6))", 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.572245+0000", 2024-04-18T21:52:31.209 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.567378608, 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31376227600000001, 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572245+0000", 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572245+0000", 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572245+0000", 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2e-07 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572283+0000", 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7953000000000001e-05 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572307+0000", 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3626e-05 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.572323+0000", 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6367000000000001e-05 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.880773+0000", 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30845004199999998 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.880788+0000", 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4749999999999999e-05 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.885929+0000", 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051404720000000001 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886007+0000", 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8646000000000006e-05 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:535 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~802816] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7411387380000001, 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60500271100000003, 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.214 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 535 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398487+0000", 2024-04-18T21:52:31.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0650000000000001e-06 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399399+0000", 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091160899999999997 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399417+0000", 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7915999999999999e-05 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399426+0000", 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7070000000000004e-06 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.485946+0000", 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086519677000000003 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.486140+0000", 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019389 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.488798+0000", 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002657707 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.600548+0000", 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11175003 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003305+0000", 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40275720799999998 2024-04-18T21:52:31.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003488+0000", 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000182902 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:536 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~638976] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7361610809999997, 2024-04-18T21:52:31.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60040761799999998, 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 536 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.09e-07 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404080+0000", 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00061689199999999996 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404092+0000", 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2055000000000001e-05 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404101+0000", 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5650000000000002e-06 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491527+0000", 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087426312000000006 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491697+0000", 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017024799999999999 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.493862+0000", 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021641299999999998 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700761+0000", 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20689907199999999 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003718+0000", 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30295725099999998 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003870+0000", 2024-04-18T21:52:31.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015228400000000001 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:537 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~131072] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7351914199999996, 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59990886700000001, 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.225 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 537 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:31.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404433+0000", 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.44e-07 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404610+0000", 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017705100000000001 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404620+0000", 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8050000000000001e-06 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404625+0000", 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5029999999999997e-06 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495333+0000", 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.090708141000000006 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495532+0000", 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001983 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.496836+0000", 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013041789999999999 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700772+0000", 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20393593900000001 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004176+0000", 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30340381399999999 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004341+0000", 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016549100000000001 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.230 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:538 3.10s0 3:0e754656:::smithi09731216-19:head [setxattr _header (57) in=7b] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.734989433, 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70001739100000004, 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 538 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404635+0000", 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6699999999999999e-07 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404637+0000", 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.683e-06 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404642+0000", 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.967e-06 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404647+0000", 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.25e-06 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.497929+0000", 2024-04-18T21:52:31.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.093281966999999993 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498095+0000", 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016633800000000001 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498905+0000", 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081039199999999995 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700775+0000", 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20186959700000001 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104337+0000", 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40356159400000002 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104652+0000", 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031513599999999999 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:539 3.10s0 3:0e754656:::smithi09731216-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7349713590000002, 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90384237700000003, 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 539 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404653+0000", 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0600000000000001e-07 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404655+0000", 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.95e-06 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404662+0000", 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9530000000000002e-06 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404666+0000", 2024-04-18T21:52:31.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4089999999999996e-06 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499383+0000", 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094717379000000004 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499540+0000", 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015672099999999999 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.105824+0000", 2024-04-18T21:52:31.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60628387699999997 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.106088+0000", 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026422700000000002 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.308495+0000", 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20240645500000001 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 69/43 ECSubWriteReply(tid=52, last_complete=69'43, committed=1, applied=1))", 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.693634+0000", 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.4459895469999999, 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30871895199999999, 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.693634+0000", 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.693634+0000", 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.693635+0000", 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.693641+0000", 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4169999999999997e-06 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.693659+0000", 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7376000000000001e-05 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.693672+0000", 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3803e-05 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.002288+0000", 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30861545000000001 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.002353+0000", 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5205999999999999e-05 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 73/72 [PushReplyOp(3:08dd949e:::smithi09731216-47:head)])", 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.437237+0000", 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.70238680399999998, 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39837960100000003, 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:31.244 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.437237+0000", 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.437237+0000", 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.437238+0000", 2024-04-18T21:52:31.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7000000000000003e-07 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.437245+0000", 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.844e-06 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.437261+0000", 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5994e-05 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:31.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.537349+0000", 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100087862 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.835540+0000", 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29819115800000001 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.835616+0000", 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6773000000000005e-05 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.247 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.248 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.248 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.248 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.248 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:31.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:31.249 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:31.249 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:31.249 INFO:tasks.rados.rados.0.smithi097.stdout:235: finishing copy_from racing read to smithi09731216-15 2024-04-18T21:52:31.249 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:31.250 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:31.250 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:31.250 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:31.251 INFO:tasks.rados.rados.0.smithi097.stdout:236: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-18T21:52:31.251 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:31.196+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:31.265 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:31.266 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:31.281 INFO:tasks.rados.rados.0.smithi097.stdout:231: finishing write tid 2 to smithi09731216-2 2024-04-18T21:52:31.281 INFO:tasks.rados.rados.0.smithi097.stdout:231: finishing write tid 3 to smithi09731216-2 2024-04-18T21:52:31.281 INFO:tasks.rados.rados.0.smithi097.stdout:231: finishing write tid 4 to smithi09731216-2 2024-04-18T21:52:31.297 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:31.291+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:31.345 INFO:tasks.rados.rados.0.smithi097.stdout:229: finishing write tid 6 to smithi09731216-48 2024-04-18T21:52:31.345 INFO:tasks.rados.rados.0.smithi097.stdout:229: oid 48 updating version 0 to 60 2024-04-18T21:52:31.345 INFO:tasks.rados.rados.0.smithi097.stdout:229: oid 48 updating version 60 to 63 2024-04-18T21:52:31.345 INFO:tasks.rados.rados.0.smithi097.stdout:229: oid 48 version 63 is already newer than 62 2024-04-18T21:52:31.345 INFO:tasks.rados.rados.0.smithi097.stdout:229: oid 48 version 63 is already newer than 61 2024-04-18T21:52:31.345 INFO:tasks.rados.rados.0.smithi097.stdout:229: oid 48 version 63 is already newer than 59 2024-04-18T21:52:31.345 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 48 v 63 (ObjNum 98 snap 19 seq_num 98) dirty exists 2024-04-18T21:52:31.345 INFO:tasks.rados.rados.0.smithi097.stdout:229: left oid 48 (ObjNum 98 snap 19 seq_num 98) 2024-04-18T21:52:31.379 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:31.379 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:31.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.379 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.10s7 74/72 ECSubWrite(tid=69, reqid=client.4509.0:588, at_version=74'48, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:31.348789+0000", 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.031018981000000001, 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031037408999999998, 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.348789+0000", 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.348789+0000", 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.348790+0000", 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3600000000000002e-07 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.348820+0000", 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0312999999999999e-05 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:31.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.348836+0000", 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5920999999999999e-05 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 1 2024-04-18T21:52:31.382 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:31.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:31.397 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:31.395+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:31.492 INFO:tasks.rados.rados.0.smithi097.stdout:235: finishing copy_from to smithi09731216-15 2024-04-18T21:52:31.492 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 15 v 112 (ObjNum 92 snap 18 seq_num 92) dirty exists 2024-04-18T21:52:31.497 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:31.496+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:31.518 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:31.518 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:31.518 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:31.519 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:31.519 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:31.525 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is set 2024-04-18T21:52:31.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:31.567 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:31.567 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:31.596 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:31.596 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:31.598 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:31.595+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:31.642 INFO:tasks.rados.rados.0.smithi097.stdout:231: finishing write tid 5 to smithi09731216-2 2024-04-18T21:52:31.642 INFO:tasks.rados.rados.0.smithi097.stdout:231: oid 2 updating version 0 to 20 2024-04-18T21:52:31.642 INFO:tasks.rados.rados.0.smithi097.stdout:231: oid 2 version 20 is already newer than 19 2024-04-18T21:52:31.643 INFO:tasks.rados.rados.0.smithi097.stdout:231: oid 2 version 20 is already newer than 17 2024-04-18T21:52:31.643 INFO:tasks.rados.rados.0.smithi097.stdout:231: oid 2 version 20 is already newer than 18 2024-04-18T21:52:31.643 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 2 v 20 (ObjNum 99 snap 19 seq_num 99) dirty exists 2024-04-18T21:52:31.643 INFO:tasks.rados.rados.0.smithi097.stdout:231: left oid 2 (ObjNum 99 snap 19 seq_num 99) 2024-04-18T21:52:31.658 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:31.658 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:31.661 INFO:tasks.rados.rados.0.smithi097.stdout:229: done (7 left) 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:231: done (6 left) 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:233: done (5 left) 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:234: done (4 left) 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:235: done (3 left) 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:236: done (2 left) 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:237: done (1 left) 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:238: rollback oid 38 current snap is 19 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 38 to 8 2024-04-18T21:52:31.662 INFO:tasks.rados.rados.0.smithi097.stdout:239: delete oid 18 current snap is 19 2024-04-18T21:52:31.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:31.677 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:31.678 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:31.699 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:31.695+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:31.704 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:31.704 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:31.729 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:31.729 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:31.756 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:31.757 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:31.812 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:31.813 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:31.821 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:31.819+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:31.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:31.900 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:31.899+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:31.950 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:31.950 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:31.965 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:31.969 INFO:tasks.rados.rados.0.smithi097.stdout:239: done (2 left) 2024-04-18T21:52:31.969 INFO:tasks.rados.rados.0.smithi097.stdout:240: copy_from oid 42 from oid 3 current snap is 19 2024-04-18T21:52:31.969 INFO:tasks.rados.rados.0.smithi097.stdout:241: rollback oid 15 current snap is 19 2024-04-18T21:52:31.969 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 15 to 8 2024-04-18T21:52:31.970 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 16 current snap is 19 2024-04-18T21:52:31.970 INFO:tasks.rados.rados.0.smithi097.stdout:242: seq_num 103 ranges {2768896=573440,3342336=786432,4128768=376832} 2024-04-18T21:52:32.001 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:31.998+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:32.016 INFO:tasks.rados.rados.0.smithi097.stdout:242: writing smithi09731216-16 from 2768896 to 3342336 tid 1 2024-04-18T21:52:32.025 INFO:tasks.rados.rados.0.smithi097.stdout:242: writing smithi09731216-16 from 3342336 to 4128768 tid 2 2024-04-18T21:52:32.030 INFO:tasks.rados.rados.0.smithi097.stdout:242: writing smithi09731216-16 from 4128768 to 4505600 tid 3 2024-04-18T21:52:32.030 INFO:tasks.rados.rados.0.smithi097.stdout:243: rollback oid 45 current snap is 19 2024-04-18T21:52:32.030 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 45 to 14 2024-04-18T21:52:32.030 INFO:tasks.rados.rados.0.smithi097.stdout:244: snap_remove snap 18 2024-04-18T21:52:32.084 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:32.084 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:32.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:32.102 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:32.099+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:32.203 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:32.199+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:32.218 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:32.218 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:32.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:32.304 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:32.298+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:32.355 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:32.355 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:32.362 INFO:tasks.rados.rados.0.smithi097.stdout:244: done (6 left) 2024-04-18T21:52:32.362 INFO:tasks.rados.rados.0.smithi097.stdout:245: snap_remove snap 15 2024-04-18T21:52:32.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:32.399 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:32.399 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:52:32.399 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:32.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 11 2024-04-18T21:52:32.405 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:32.402+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:32.503 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:32.503 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:32.506 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:32.502+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:32.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:32.607 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:32.603+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:32.643 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:32.643 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:32.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:32.707 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:32.703+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:32.725 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-18T21:52:32.723+0000 7f0412e7b640 -1 mon.a@0(leader).osd e76 definitely_dead 0 2024-04-18T21:52:32.783 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:32.783 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:32.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:32.808 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:32.802+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:32.909 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:32.907+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:32.924 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:32.924 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:32.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:33.010 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:33.008+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:33.058 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:33.058 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:33.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:33.116 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:33.116+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:33.212 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:33.206+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:33.242 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:33.242 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:33.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:33.313 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:33.312+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:33.362 INFO:teuthology.orchestra.run.smithi097.stderr:marked out osd.11. 2024-04-18T21:52:33.370 INFO:tasks.rados.rados.0.smithi097.stdout:245: done (6 left) 2024-04-18T21:52:33.370 INFO:tasks.rados.rados.0.smithi097.stdout:246: copy_from oid 11 from oid 20 current snap is 19 2024-04-18T21:52:33.370 INFO:tasks.rados.rados.0.smithi097.stdout:238: finishing rollback tid 0 to smithi09731216-38 2024-04-18T21:52:33.370 INFO:tasks.rados.rados.0.smithi097.stdout:247: delete oid 25 current snap is 19 2024-04-18T21:52:33.370 INFO:tasks.rados.rados.0.smithi097.stdout:241: finishing rollback tid 0 to smithi09731216-15 2024-04-18T21:52:33.370 INFO:tasks.rados.rados.0.smithi097.stdout:240: finishing copy_from to smithi09731216-42 2024-04-18T21:52:33.370 INFO:tasks.rados.rados.0.smithi097.stdout:240: got expected ENOENT (src dne) 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:240: finishing copy_from racing read to smithi09731216-42 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:243: finishing rollback tid 0 to smithi09731216-45 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:243: finishing rollback tid 1 to smithi09731216-45 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:243: finishing rollback tid 2 to smithi09731216-45 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 45 v 65 (ObjNum 84 snap 14 seq_num 84) dirty exists 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 35 v 22 (ObjNum 70 snap 4 seq_num 70) dirty exists 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:241: finishing rollback tid 1 to smithi09731216-15 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:241: finishing rollback tid 2 to smithi09731216-15 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 15 v 129 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-18T21:52:33.371 INFO:tasks.rados.rados.0.smithi097.stdout:242: finishing write tid 1 to smithi09731216-16 2024-04-18T21:52:33.387 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:33.388 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:33.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:33.414 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:33.410+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:33.515 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:33.510+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:33.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:52:33.573 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:33.573 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:33.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:33.637 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:33.635+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:33.716 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:33.711+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:33.762 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:33.762 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:33.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:33.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:33.817 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:33.816+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:33.919 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:33.916+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:33.944 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:33.944 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:33.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:33.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:33.961 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:33.979 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:33.979 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:33.995 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:33.996 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:34.020 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:34.020+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:34.026 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:34.026 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:34.046 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:34.046 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:34.067 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:34.067 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:34.074 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:34.074 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:34.074 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:34.074 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:34.087 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:34.088 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:34.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:34.109 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:34.109 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:34.132 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:34.132 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:34.132 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:34.127+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:34.153 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:34.153 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:34.174 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:34.174 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:34.196 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:34.196 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:34.223 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:34.220+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:34.226 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:34.226 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:34.226 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:34.226 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:34.226 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:34.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:34.327 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:34.322+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:34.364 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:94 3.0s0 3:033c99c3:::smithi09731216-32:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.859759992000001, 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40893576500000001, 2024-04-18T21:52:34.417 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555853+0000", 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1889999999999999e-06 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556617+0000", 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076362200000000004 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556649+0000", 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2478999999999999e-05 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556665+0000", 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5837000000000001e-05 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556826+0000", 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016137200000000001 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.557241+0000", 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00041440699999999999 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.559642+0000", 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024011430000000001 2024-04-18T21:52:34.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563415+0000", 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003773233 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964577+0000", 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40116224099999997 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964786+0000", 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000208242 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:95 3.0s0 3:033c99c3:::smithi09731216-32:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.858906894999997, 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858905099999998, 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.423 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 95 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556705+0000", 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6360000000000001e-06 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556715+0000", 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.066e-05 2024-04-18T21:52:34.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556734+0000", 2024-04-18T21:52:34.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8323999999999999e-05 2024-04-18T21:52:34.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556745+0000", 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1240000000000001e-05 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561327+0000", 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0045823330000000001 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561583+0000", 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000255732 2024-04-18T21:52:34.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.562607+0000", 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010240990000000001 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563867+0000", 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012605439999999999 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965078+0000", 2024-04-18T21:52:34.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40121052699999998 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965292+0000", 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021395600000000001 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:96 3.0s0 3:033c99c3:::smithi09731216-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:34.429 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.858809352999998, 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42323290800000002, 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 96 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556802+0000", 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7400000000000001e-06 2024-04-18T21:52:34.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556808+0000", 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.993e-06 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556828+0000", 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9364e-05 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.656965+0000", 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10013702300000001 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762039+0000", 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105074903 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762350+0000", 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031089399999999998 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967780+0000", 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20542997700000001 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.972542+0000", 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004761932 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.979990+0000", 2024-04-18T21:52:34.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0074475289999999996 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.980033+0000", 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3553000000000001e-05 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:222 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:34.435 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.402727763999998, 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30521668800000001, 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 222 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012885+0000", 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7760000000000002e-06 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012901+0000", 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6563e-05 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012934+0000", 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2719999999999998e-05 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012952+0000", 2024-04-18T21:52:34.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7730000000000001e-05 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013179+0000", 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022670500000000001 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013200+0000", 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1466000000000001e-05 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114024+0000", 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100823616 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114179+0000", 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015544500000000001 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.318099+0000", 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203919667 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.440 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:223 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.402607334999999, 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40441434900000001, 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.441 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 223 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:34.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013004+0000", 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5400000000000001e-06 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013015+0000", 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0519999999999999e-05 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013077+0000", 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2848999999999994e-05 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013098+0000", 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1083000000000001e-05 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013359+0000", 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000260408 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013383+0000", 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3867e-05 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114847+0000", 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10146374399999999 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114996+0000", 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149865 2024-04-18T21:52:34.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.417417+0000", 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30242047300000002 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:224 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.402477243, 2024-04-18T21:52:34.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40564824999999999, 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 224 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:34.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013134+0000", 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.387e-06 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013144+0000", 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0273e-05 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013166+0000", 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1857000000000001e-05 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013182+0000", 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5645999999999999e-05 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113542+0000", 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10035996599999999 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113552+0000", 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0492999999999999e-05 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215576+0000", 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102023869 2024-04-18T21:52:34.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215730+0000", 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000153635 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.418781+0000", 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203051124 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.451 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:302 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.353958921999997, 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.406305586, 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 302 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.452 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061654+0000", 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061669+0000", 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4637000000000001e-05 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061699+0000", 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9791e-05 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061714+0000", 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5441999999999999e-05 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061811+0000", 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7153999999999998e-05 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061819+0000", 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9279999999999993e-06 2024-04-18T21:52:34.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064181+0000", 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002361447 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064306+0000", 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012514399999999999 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.065234+0000", 2024-04-18T21:52:34.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092848599999999996 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.165102+0000", 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.099867808000000002 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467877+0000", 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.302775085 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467956+0000", 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9381000000000005e-05 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:303 3.0s0 3:00e42e84:::smithi09731216-41:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.353859532000001, 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50679023400000001, 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.458 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 303 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061751+0000", 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5499999999999998e-07 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061754+0000", 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2339999999999999e-06 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061763+0000", 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3950000000000007e-06 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061769+0000", 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9730000000000002e-06 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061904+0000", 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013470999999999999 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061911+0000", 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2200000000000003e-06 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163859+0000", 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101947416 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163977+0000", 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117971 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.164692+0000", 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071513499999999999 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267070+0000", 2024-04-18T21:52:34.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102378067 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568454+0000", 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.301384333 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568541+0000", 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6224999999999995e-05 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.464 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:304 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.353817059999997, 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60693648, 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 304 2024-04-18T21:52:34.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3000000000000001e-07 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061797+0000", 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3680000000000002e-06 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061804+0000", 2024-04-18T21:52:34.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0809999999999996e-06 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061809+0000", 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5280000000000003e-06 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061950+0000", 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000140529 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:34.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061958+0000", 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5329999999999994e-06 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166016+0000", 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104057925 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166118+0000", 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010206500000000001 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166690+0000", 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057174600000000004 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267073+0000", 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100382727 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668597+0000", 2024-04-18T21:52:34.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401524149 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668729+0000", 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132399 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:332 3.0s0 3:00b6e030:::smithi09731216-21:head [copy-from ver 39] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:34.471 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 35.209658081000001, 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.72786893100000005, 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 332 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205954+0000", 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4370000000000001e-06 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205969+0000", 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4516e-05 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206001+0000", 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.252e-05 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206022+0000", 2024-04-18T21:52:34.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.084e-05 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306220+0000", 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100197961 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306483+0000", 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026249200000000002 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.726594+0000", 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42011143499999998 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.730261+0000", 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036666060000000002 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933714+0000", 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203453467 2024-04-18T21:52:34.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933821+0000", 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010665700000000001 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:333 3.0s0 3:00b6e030:::smithi09731216-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 35.209553108000001, 2024-04-18T21:52:34.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52393701999999998, 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 333 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206058+0000", 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.426e-06 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206065+0000", 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4799999999999998e-06 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206085+0000", 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0089999999999999e-05 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206101+0000", 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.643e-05 2024-04-18T21:52:34.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306839+0000", 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100738203 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306987+0000", 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014755599999999999 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729615+0000", 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42262781100000002 2024-04-18T21:52:34.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729821+0000", 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020588799999999999 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729994+0000", 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000173136 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=46, reqid=client.4509.0:360, at_version=47'10, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.591070338000002, 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40258300600000002, 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824542+0000", 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.193e-06 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824777+0000", 2024-04-18T21:52:34.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234878 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824808+0000", 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1633e-05 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924923+0000", 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10011492700000001 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.025179+0000", 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100256004 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.025197+0000", 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8326e-05 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227086+0000", 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201888453 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227122+0000", 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6591999999999998e-05 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=47, reqid=client.4509.0:365, at_version=47'11, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.490643603999999, 2024-04-18T21:52:34.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218968200000002, 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:34.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924967+0000", 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.088e-06 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925088+0000", 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120229 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925104+0000", 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6804e-05 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925115+0000", 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0516999999999999e-05 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.026507+0000", 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101391776 2024-04-18T21:52:34.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.026542+0000", 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5126999999999998e-05 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227138+0000", 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20059612399999999 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227156+0000", 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8017e-05 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=48, reqid=client.4509.0:366, at_version=47'12, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.490475965000002, 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30205703299999997, 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.492 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2500000000000001e-07 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925294+0000", 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016000000000000001 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925308+0000", 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4144999999999999e-05 2024-04-18T21:52:34.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925319+0000", 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.008e-05 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.027610+0000", 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102291967 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.027638+0000", 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7453e-05 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227173+0000", 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19953516700000001 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227191+0000", 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7796e-05 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.496 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=49, reqid=client.4509.0:367, at_version=47'13, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.490278183000001, 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30187523100000002, 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:34.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4700000000000002e-07 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925351+0000", 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8998e-05 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925362+0000", 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1302e-05 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925370+0000", 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6680000000000001e-06 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.028840+0000", 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10347008100000001 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.028867+0000", 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6982999999999999e-05 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227194+0000", 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19832709500000001 2024-04-18T21:52:34.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227207+0000", 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2656999999999999e-05 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=50, reqid=client.4509.0:361, at_version=47'14, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 30.490227988000001, 2024-04-18T21:52:34.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30184082099999998, 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:34.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6600000000000002e-07 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925457+0000", 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4703000000000006e-05 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925469+0000", 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1932e-05 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925477+0000", 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1529999999999997e-06 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.029353+0000", 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10387573999999999 2024-04-18T21:52:34.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.029380+0000", 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7033e-05 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227210+0000", 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19783015700000001 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227223+0000", 2024-04-18T21:52:34.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2737e-05 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:430 3.0s0 3:00b6e030:::smithi09731216-21:head [copy-from ver 39] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 19.969883587999998, 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71883465800000002, 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.506 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 430 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445730+0000", 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7629999999999998e-06 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445749+0000", 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8960000000000001e-05 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445786+0000", 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7049999999999999e-05 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445804+0000", 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8236e-05 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445984+0000", 2024-04-18T21:52:34.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017996200000000001 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446012+0000", 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7554000000000001e-05 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.515978+0000", 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069965816 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516184+0000", 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020612199999999999 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.970820+0000", 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45463621900000001 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.978038+0000", 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0072179430000000001 2024-04-18T21:52:34.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.164404+0000", 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18636640400000001 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.164561+0000", 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015662900000000001 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:431 3.0s0 3:00b6e030:::smithi09731216-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 19.969757835999999, 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52894810699999995, 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 431 2024-04-18T21:52:34.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445853+0000", 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.437e-06 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445863+0000", 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4769999999999992e-06 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445885+0000", 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2093e-05 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445901+0000", 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5826999999999999e-05 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446183+0000", 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028208399999999999 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446201+0000", 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8133000000000001e-05 2024-04-18T21:52:34.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516497+0000", 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.070296147000000003 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516639+0000", 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000141533 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974265+0000", 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45762602699999999 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974495+0000", 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022996099999999999 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974800+0000", 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030538800000000002 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.518 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:560 3.0s0 3:00e42e84:::smithi09731216-41:head [copy-from ver 4] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.397183364, 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41315372, 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.519 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 560 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018429+0000", 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.357e-06 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018443+0000", 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4491999999999999e-05 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018468+0000", 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5060000000000001e-05 2024-04-18T21:52:34.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018488+0000", 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9259e-05 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.118707+0000", 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10021904099999999 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.118865+0000", 2024-04-18T21:52:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015874199999999999 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.226410+0000", 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107544398 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.229100+0000", 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026898130000000001 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.431493+0000", 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20239311800000001 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.431580+0000", 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7440000000000003e-05 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 73/31 ECSubWriteReply(tid=52, last_complete=73'30, committed=1, applied=1))", 2024-04-18T21:52:34.524 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.534395+0000", 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 3.8812144559999999, 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30324524800000002, 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534395+0000", 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534395+0000", 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534396+0000", 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.02e-07 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534402+0000", 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6799999999999998e-06 2024-04-18T21:52:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534416+0000", 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3848e-05 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.634508+0000", 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10009191000000001 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.837557+0000", 2024-04-18T21:52:34.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20304905200000001 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.837641+0000", 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3856000000000005e-05 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.528 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:34.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:34.530 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:34.528+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:34.631 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:34.628+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:34.701 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:34.702 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:34.702 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:34.702 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:34.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:34.733 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:34.730+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:34.833 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:34.830+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:34.838 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:34.839 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:34.839 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:34.839 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:34.839 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:34.853 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:34.937 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:34.931+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:34.977 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:34.977 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:83 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.925906443999999, 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28825372900000001, 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.978 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 83 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:34.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2e-07 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050828+0000", 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3929999999999998e-06 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050832+0000", 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4909999999999999e-06 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050836+0000", 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5219999999999999e-06 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052455+0000", 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016189710000000001 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052552+0000", 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7213999999999995e-05 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052983+0000", 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043119300000000002 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138342+0000", 2024-04-18T21:52:34.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.085359090999999998 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.338942+0000", 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200600308 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339078+0000", 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136026 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:84 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.925877786000001, 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59192865299999997, 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 84 2024-04-18T21:52:34.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050854+0000", 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1399999999999998e-07 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050855+0000", 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5719999999999999e-06 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050860+0000", 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-04-18T21:52:34.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050864+0000", 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5750000000000001e-06 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138083+0000", 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087218981000000001 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138226+0000", 2024-04-18T21:52:34.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000143545 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339757+0000", 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20153072 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339856+0000", 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9247999999999996e-05 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642782+0000", 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30292579400000003 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.es0 30/26 ECSubRead(tid=10, to_read={3:77cf8780:::smithi09731216-28:head=0,4096,0}, subchunks={3:77cf8780:::smithi09731216-28:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.636717212999997, 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218228800000002, 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.989 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340093+0000", 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8665000000000007e-05 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440291+0000", 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100198545 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641845+0000", 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20155352000000001 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642196+0000", 2024-04-18T21:52:34.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035155800000000002 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.cs7 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.213176034, 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30181939000000002, 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763557+0000", 2024-04-18T21:52:34.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.776e-06 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763600+0000", 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3120000000000001e-05 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763635+0000", 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5199e-05 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.863748+0000", 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100112399 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964004+0000", 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10025664500000001 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964030+0000", 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5299999999999998e-05 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065244+0000", 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101214472 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065374+0000", 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000130479 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=20, last_complete=35'12, committed=1, applied=1))", 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.598394243999998, 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225695955, 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378337+0000", 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378338+0000", 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3499999999999996e-07 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378343+0000", 2024-04-18T21:52:34.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0669999999999999e-06 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378356+0000", 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3264e-05 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378367+0000", 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1069e-05 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.603689+0000", 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22532233900000001 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604033+0000", 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034358099999999998 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=21, last_complete=35'13, committed=1, applied=1))", 2024-04-18T21:52:35.001 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.598353889000002, 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22690856100000001, 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378378+0000", 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378380+0000", 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7190000000000001e-06 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378389+0000", 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378395+0000", 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5210000000000002e-06 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604185+0000", 2024-04-18T21:52:35.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22579012100000001 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.605286+0000", 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011008979999999999 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:230 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:35.005 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.963462540000002, 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.312981385, 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 230 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013270+0000", 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5300000000000002e-07 2024-04-18T21:52:35.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013275+0000", 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013291+0000", 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5466000000000001e-05 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013303+0000", 2024-04-18T21:52:35.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2e-05 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013750+0000", 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044700300000000002 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013767+0000", 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7263999999999999e-05 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225043+0000", 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21127590299999999 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225199+0000", 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155716 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.326250+0000", 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10105134 2024-04-18T21:52:35.010 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:315 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.883954463000002, 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70415003200000004, 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.011 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 315 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:35.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092778+0000", 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.725e-06 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092785+0000", 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7950000000000002e-06 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092802+0000", 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6852999999999999e-05 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092810+0000", 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7950000000000008e-06 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152681+0000", 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059871316000000001 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152755+0000", 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4179999999999998e-05 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153317+0000", 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00056207900000000005 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253434+0000", 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100116332 2024-04-18T21:52:35.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796715+0000", 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54328157899999996 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796927+0000", 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021137799999999999 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:316 3.es0 3:7b3c3586:::smithi09731216-12:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.883905718999998, 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70474535999999999, 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 316 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.017 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092826+0000", 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5999999999999999e-07 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092827+0000", 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714e-06 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092833+0000", 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0989999999999998e-06 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092836+0000", 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.056e-06 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153954+0000", 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061118211999999998 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154059+0000", 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010548399999999999 2024-04-18T21:52:35.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154712+0000", 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065292300000000002 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253436+0000", 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098723985 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797355+0000", 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.543919245 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797571+0000", 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021528199999999999 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:317 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.883865315999998, 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70519442399999999, 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 317 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000002e-07 2024-04-18T21:52:35.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092868+0000", 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1440000000000001e-06 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092873+0000", 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0860000000000001e-06 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192949+0000", 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007585099999999 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252838+0000", 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059889100000000001 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252927+0000", 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8651000000000002e-05 2024-04-18T21:52:35.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253260+0000", 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033289600000000001 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.353758+0000", 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100498198 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797947+0000", 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.444189102 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.798060+0000", 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113109 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.028 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=6, reqid=client.4509.0:267, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.271265819999996, 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.305911983, 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705468+0000", 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4229999999999999e-06 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705683+0000", 2024-04-18T21:52:35.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021499900000000001 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705719+0000", 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6758999999999998e-05 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805843+0000", 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10012386299999999 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806004+0000", 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160722 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806040+0000", 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6335999999999999e-05 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011325+0000", 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20528459499999999 2024-04-18T21:52:35.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011377+0000", 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2286e-05 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:474 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~688128] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 15.971596996000001, 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31877274300000003, 2024-04-18T21:52:35.033 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 474 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005137+0000", 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000001e-06 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005928+0000", 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00079063400000000002 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005960+0000", 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2727e-05 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005976+0000", 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5937e-05 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014929+0000", 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0089524860000000008 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.015108+0000", 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000179366 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.017762+0000", 2024-04-18T21:52:35.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026539900000000002 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106305+0000", 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.088543338999999999 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.323764+0000", 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.217458182 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.323907+0000", 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014321200000000001 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:475 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~540672] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 15.962653369, 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31080646899999997, 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.039 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 475 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014079+0000", 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.72e-06 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014732+0000", 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065293199999999995 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014757+0000", 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4352999999999999e-05 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014770+0000", 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.324e-05 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.020116+0000", 2024-04-18T21:52:35.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053463540000000002 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.020404+0000", 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028811800000000002 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.022644+0000", 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022399899999999999 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106309+0000", 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.083664469000000005 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.324710+0000", 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21840135099999999 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.324884+0000", 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017394200000000001 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.044 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:476 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~229376] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 15.958326159, 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30710270099999998, 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.045 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 476 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018408+0000", 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9660000000000001e-06 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018913+0000", 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00050535599999999999 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018956+0000", 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2740000000000001e-05 2024-04-18T21:52:35.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019019+0000", 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2694000000000007e-05 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.023815+0000", 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047959430000000004 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.024087+0000", 2024-04-18T21:52:35.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027221100000000001 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.025513+0000", 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014256360000000001 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106311+0000", 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.080798549999999997 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.325343+0000", 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21903152300000001 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.325508+0000", 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016508199999999999 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:477 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:35.050 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 15.957672931999999, 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40712708600000003, 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 477 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019060+0000", 2024-04-18T21:52:35.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7859999999999999e-06 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019070+0000", 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0345e-05 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019098+0000", 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7392000000000001e-05 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019115+0000", 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7765e-05 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.026938+0000", 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078220470000000004 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.027160+0000", 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000222625 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.027978+0000", 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081757100000000001 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106314+0000", 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.078336021000000006 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.426038+0000", 2024-04-18T21:52:35.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31972404500000001 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.426185+0000", 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000147489 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:478 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 15.957588004, 2024-04-18T21:52:35.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.709574606, 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 478 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019144+0000", 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.206e-06 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019151+0000", 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4819999999999999e-06 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019170+0000", 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8906000000000002e-05 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019185+0000", 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5116999999999999e-05 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.028449+0000", 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0092640520000000001 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.028689+0000", 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023998100000000001 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.427127+0000", 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39843774999999998 2024-04-18T21:52:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.427384+0000", 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025700499999999998 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.728718+0000", 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30133410700000002 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:562 3.es0 3:77cf8780:::smithi09731216-28:head [delete] snapc 14=[14,11,10,a,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.018554+0000", 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.9581773680000003, 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39250216100000002, 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 562 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.062 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018554+0000", 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018554+0000", 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018557+0000", 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.836e-06 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018568+0000", 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1610000000000001e-05 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018600+0000", 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2060000000000001e-05 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018617+0000", 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7033000000000001e-05 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018756+0000", 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013826299999999999 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.019238+0000", 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00048212599999999998 2024-04-18T21:52:35.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.021858+0000", 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002619864 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.024162+0000", 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023041699999999999 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.410181+0000", 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38601886000000002 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.411056+0000", 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00087533900000000004 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:578 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.743123+0000", 2024-04-18T21:52:35.067 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.23360804, 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40546945600000001, 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 578 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743123+0000", 2024-04-18T21:52:35.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743123+0000", 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743125+0000", 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9259999999999999e-06 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743133+0000", 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6580000000000002e-06 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743151+0000", 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8417e-05 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743161+0000", 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0237999999999999e-05 2024-04-18T21:52:35.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743235+0000", 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3946e-05 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743525+0000", 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028927899999999998 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:31.148593+0000", 2024-04-18T21:52:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40506799199999999 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:579 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.2335463789999999, 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40650074000000003, 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.072 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 579 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:52:35.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743186+0000", 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.16e-07 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743190+0000", 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5480000000000003e-06 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743199+0000", 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4570000000000005e-06 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743205+0000", 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7959999999999998e-06 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.844198+0000", 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100992966 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.844801+0000", 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060208199999999999 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:31.149685+0000", 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.304884975 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:580 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.2335154490000004, 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40696587899999997, 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 580 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:52:35.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4799999999999998e-07 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743220+0000", 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5829999999999998e-06 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743227+0000", 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.379e-06 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743233+0000", 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9220000000000002e-06 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.937914+0000", 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19468091200000001 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.938494+0000", 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000580282 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:31.150182+0000", 2024-04-18T21:52:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21168725299999999 2024-04-18T21:52:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.081 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.081 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:35.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:35.083 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:35.079+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:35.183 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:35.184+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:35.253 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:35.253 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:35.253 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:35.253 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:35.267 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:35.313 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:35.311+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:35.385 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:35.383+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:35.393 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:35.393 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:35.393 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:35.394 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:35.394 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:35.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:35.482 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:35.478+0000 7f3d90efa640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:35.483 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:35.478+0000 7f3d90efa640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:35.491 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:35.482+0000 7f3d90efa640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:35.492 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:35.482+0000 7f3d90efa640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:35.492 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:35.486+0000 7f3d90efa640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:35.497 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:35.495+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:35.584 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:35.584 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:35.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.8s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.816378663000002, 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51395767000000003, 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:52:35.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764361+0000", 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.61e-06 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764368+0000", 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1600000000000001e-06 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764389+0000", 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1073e-05 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764409+0000", 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0006000000000001e-05 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866089+0000", 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101679558 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866221+0000", 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132022 2024-04-18T21:52:35.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278264+0000", 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41204366100000001 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278317+0000", 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2580000000000001e-05 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:103 3.18s0 3:1b279e7b:::smithi09731216-35:head [append 0~573440,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.599720198999997, 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6106588159999999, 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 103 2024-04-18T21:52:35.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981021+0000", 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5470000000000001e-06 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981778+0000", 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00075675099999999995 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981811+0000", 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3200000000000001e-05 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981832+0000", 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0703000000000002e-05 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.277997+0000", 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29616472700000002 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278003+0000", 2024-04-18T21:52:35.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3269999999999997e-06 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591581+0000", 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.313578111 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591677+0000", 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5450000000000003e-05 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:104 3.18s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.598855344999997, 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714877378, 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981885+0000", 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0260000000000002e-06 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981895+0000", 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7969999999999995e-06 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981918+0000", 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3116e-05 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981931+0000", 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2850000000000001e-05 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278086+0000", 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29615502199999999 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278088+0000", 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6230000000000001e-06 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696611+0000", 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418522515 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696760+0000", 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149429 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.598791038999998, 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9142408849999999, 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.599 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981948+0000", 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981952+0000", 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8739999999999997e-06 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981963+0000", 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.133e-05 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981971+0000", 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2900000000000002e-06 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278161+0000", 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29618945299999999 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278163+0000", 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.734e-06 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696819+0000", 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418656006 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.697058+0000", 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023910700000000001 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.896188+0000", 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19913024800000001 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.604 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:227 3.19s0 3:9941efaf:::smithi09731216-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.567578759, 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50662614500000003, 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 227 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013160+0000", 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1299999999999999e-07 2024-04-18T21:52:35.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013167+0000", 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9670000000000004e-06 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013187+0000", 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9726000000000001e-05 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013202+0000", 2024-04-18T21:52:35.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5483000000000001e-05 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013460+0000", 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025794499999999998 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013468+0000", 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8460000000000008e-06 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.315837+0000", 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30236893999999997 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.316131+0000", 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029429400000000001 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.519785+0000", 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20365413099999999 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.609 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:318 3.19s0 3:9da3af1e:::smithi09731216-23:head [delete] snapc 5=[2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 37.387865216999998, 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60663671399999997, 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 318 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192874+0000", 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2699999999999998e-07 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192878+0000", 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0239999999999999e-06 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192886+0000", 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2549999999999997e-06 2024-04-18T21:52:35.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192890+0000", 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.357e-06 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372031+0000", 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.179140625 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372092+0000", 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1457999999999993e-05 2024-04-18T21:52:35.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372626+0000", 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053349599999999995 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.374211+0000", 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001584752 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799295+0000", 2024-04-18T21:52:35.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42508415500000002 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799510+0000", 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021466500000000001 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:329 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:35.615 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.357089944000002, 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60620628200000004, 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 329 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:35.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6600000000000003e-07 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223651+0000", 2024-04-18T21:52:35.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5919999999999999e-06 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223655+0000", 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9890000000000003e-06 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223658+0000", 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2600000000000001e-06 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514422+0000", 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29076426599999999 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514584+0000", 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000161275 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625434+0000", 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.110850424 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625669+0000", 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234315 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.829854+0000", 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20418589500000001 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:360 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~475136] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.216551793999997, 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98249101000000005, 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 360 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.621 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364189+0000", 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8420000000000001e-06 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364486+0000", 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029724500000000002 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364517+0000", 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0994000000000002e-05 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364530+0000", 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2330999999999999e-05 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364644+0000", 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114673 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364657+0000", 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2258e-05 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.394826+0000", 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.030169702 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395495+0000", 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00066827599999999998 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.821805+0000", 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42631051199999997 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.822025+0000", 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000220084 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824602+0000", 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025771510000000002 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944319+0000", 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11971646900000001 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346144+0000", 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401824764 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346677+0000", 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053370900000000005 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:361 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~622592 in=622592b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:35.627 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.211811562999998, 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2802270259999999, 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 361 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:35.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368928+0000", 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6559999999999999e-06 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369217+0000", 2024-04-18T21:52:35.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028880699999999997 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369238+0000", 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0914999999999999e-05 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369247+0000", 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1910000000000006e-06 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369333+0000", 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5482000000000006e-05 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369345+0000", 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2147000000000001e-05 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395635+0000", 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026290322000000001 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395761+0000", 2024-04-18T21:52:35.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126348 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837828+0000", 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44206649199999998 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837999+0000", 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017102900000000001 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.840298+0000", 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022987670000000002 2024-04-18T21:52:35.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944332+0000", 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10403438199999999 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648919+0000", 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70458679599999996 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649154+0000", 2024-04-18T21:52:35.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234692 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:362 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~147456 in=147456b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.210831937000002, 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.279510908, 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.634 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 362 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369907+0000", 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0529999999999999e-06 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369994+0000", 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6688999999999996e-05 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370003+0000", 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3789999999999995e-06 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370010+0000", 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9229999999999996e-06 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370098+0000", 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8146000000000006e-05 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370122+0000", 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3329e-05 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395809+0000", 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025687331000000001 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395945+0000", 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013612499999999999 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941837+0000", 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54589167500000002 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941940+0000", 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103435 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942633+0000", 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069312599999999996 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244973+0000", 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30233927599999999 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649234+0000", 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40426141100000001 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649417+0000", 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018301 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.640 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:363 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.210715763000003, 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.380372242, 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.641 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 363 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370023+0000", 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.75e-07 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370027+0000", 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8489999999999999e-06 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370039+0000", 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2013e-05 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370043+0000", 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6630000000000001e-06 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370169+0000", 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125506 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370180+0000", 2024-04-18T21:52:35.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0798000000000001e-05 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396197+0000", 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026017723999999999 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396320+0000", 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122905 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943149+0000", 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54682909400000002 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943272+0000", 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012215699999999999 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943899+0000", 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000627106 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244975+0000", 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30107650600000002 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.749791+0000", 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50481555700000003 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.750395+0000", 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060378899999999998 2024-04-18T21:52:35.647 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:365 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~491520] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.207106029999998, 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97351366699999997, 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.648 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 365 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373634+0000", 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.024e-06 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373873+0000", 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000238459 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373899+0000", 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6461999999999998e-05 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373912+0000", 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2823000000000001e-05 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374028+0000", 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115758 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374050+0000", 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2314e-05 2024-04-18T21:52:35.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.601683+0000", 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22763349599999999 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.602117+0000", 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043313999999999999 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828190+0000", 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226073089 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828481+0000", 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029162099999999999 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.830297+0000", 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018160229999999999 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944323+0000", 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114025979 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346866+0000", 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40254325000000002 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.347146+0000", 2024-04-18T21:52:35.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027922899999999998 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:366 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~770048] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.201487098999998, 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.067907554, 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 366 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379253+0000", 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9609999999999999e-06 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379714+0000", 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046141699999999999 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379730+0000", 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5585000000000001e-05 2024-04-18T21:52:35.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379738+0000", 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379830+0000", 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2551000000000002e-05 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379852+0000", 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.143e-05 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.604888+0000", 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225036559 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.605008+0000", 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119936 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831601+0000", 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226592508 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831778+0000", 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017783799999999999 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.834089+0000", 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023105930000000001 2024-04-18T21:52:35.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944325+0000", 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11023567500000001 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.446970+0000", 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50264567000000004 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.447159+0000", 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018811199999999999 2024-04-18T21:52:35.661 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:367 3.11s0 3:8af06b78:::smithi09731216-10:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.200982181999997, 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1678633460000001, 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 367 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4899999999999995e-07 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379760+0000", 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4170000000000001e-06 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379766+0000", 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7459999999999997e-06 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379769+0000", 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.608e-06 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379929+0000", 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015980200000000001 2024-04-18T21:52:35.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379953+0000", 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3764000000000002e-05 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707668+0000", 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32771509500000001 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:35.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707855+0000", 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186987 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835388+0000", 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127533496 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835835+0000", 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044705299999999999 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:35.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.836767+0000", 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00093163299999999996 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944330+0000", 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107563336 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547378+0000", 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60304798000000004 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547619+0000", 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024088000000000001 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:368 3.11s0 3:8af06b78:::smithi09731216-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 32.200946252000001, 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6760458580000002, 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.669 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 368 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:35.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.91e-07 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379799+0000", 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2690000000000002e-06 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379805+0000", 2024-04-18T21:52:35.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7989999999999999e-06 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379809+0000", 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4929999999999998e-06 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379996+0000", 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186959 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.380015+0000", 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9837e-05 2024-04-18T21:52:35.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707952+0000", 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32793653499999997 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.708113+0000", 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160536 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837592+0000", 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12947945499999999 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837760+0000", 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016796899999999999 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.647833+0000", 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.81007341499999996 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648038+0000", 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020446900000000001 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:05.055838+0000", 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.407799831 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=50, last_complete=47'14, committed=1, applied=1))", 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 31.353349279, 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42177490000000001, 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.675 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227389+0000", 2024-04-18T21:52:35.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8000000000000002e-07 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227392+0000", 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.57e-06 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227402+0000", 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9650000000000003e-06 2024-04-18T21:52:35.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327516+0000", 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100114438 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648849+0000", 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32133336099999998 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649164+0000", 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031428599999999999 2024-04-18T21:52:35.678 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=50, last_complete=47'14, committed=1, applied=1))", 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 31.346135364999999, 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41408854499999997, 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234603+0000", 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5199999999999998e-07 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234605+0000", 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0229999999999999e-06 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234611+0000", 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9499999999999998e-06 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.234615+0000", 2024-04-18T21:52:35.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2100000000000003e-06 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648672+0000", 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41405668699999998 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648691+0000", 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9423e-05 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.682 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=51, last_complete=47'15, committed=1, applied=1))", 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 31.344771850000001, 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41281960600000001, 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235966+0000", 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235967+0000", 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6899999999999998e-07 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235970+0000", 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3060000000000002e-06 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235981+0000", 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.083e-05 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.235987+0000", 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9309999999999996e-06 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648743+0000", 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41275640400000002 2024-04-18T21:52:35.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648786+0000", 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2666000000000001e-05 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 47/45 ECSubWriteReply(tid=52, last_complete=47'16, committed=1, applied=1))", 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 31.253145239999998, 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42417700800000002, 2024-04-18T21:52:35.686 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:35.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327593+0000", 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3099999999999999e-07 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327595+0000", 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8700000000000001e-06 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327600+0000", 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0010000000000001e-06 2024-04-18T21:52:35.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.327603+0000", 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5429999999999998e-06 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.749596+0000", 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.421992433 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.751770+0000", 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021739300000000001 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.689 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:469 3.11s0 3:8fad7925:::smithi09731216-43:head [read 0~1966080,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.010747507000001, 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50551296700000004, 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 469 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569992+0000", 2024-04-18T21:52:35.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.034e-06 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570000+0000", 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1680000000000004e-06 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570022+0000", 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2595999999999998e-05 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570033+0000", 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0858e-05 2024-04-18T21:52:35.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.571032+0000", 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00099923299999999998 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.571153+0000", 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120982 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.075504+0000", 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50435009600000003 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.693 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.694 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.694 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:35.694 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:35.694 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:35.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:35.696 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:35.690+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:35.702 INFO:tasks.rados.rados.0.smithi097.stdout:242: finishing write tid 2 to smithi09731216-16 2024-04-18T21:52:35.796 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:35.791+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:35.804 INFO:tasks.rados.rados.0.smithi097.stdout:242: finishing write tid 3 to smithi09731216-16 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:608 3.1s1 3.71ccf901 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e78)", 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.4547392539999999, 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.454777958, 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for peered", 2024-04-18T21:52:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 608 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:52:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379131+0000", 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1e-06 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379144+0000", 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2904e-05 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379188+0000", 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3909000000000003e-05 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379214+0000", 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6610000000000001e-05 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.675749+0000", 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29653525600000002 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.675755+0000", 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0129999999999999e-06 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:609 3.1s1 3.71ccf901 (undecoded) ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.4546128949999999, 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4547735669999999, 2024-04-18T21:52:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for peered", 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 609 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379255+0000", 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.474e-06 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379263+0000", 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7530000000000001e-06 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379287+0000", 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3416000000000001e-05 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379305+0000", 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.819e-05 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.775808+0000", 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39650268500000002 2024-04-18T21:52:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.775813+0000", 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8329999999999999e-06 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "pg_update_log_missing(3.13s2 epoch 79/78 rep_tid 234 entries 79'66 (0'0) error 3:c8da3643:::smithi09731216-25:head by client.4509.0:610 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:35.806043+0000", 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.027823537999999998, 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.028117352000000002, 2024-04-18T21:52:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.806043+0000", 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.806043+0000", 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.806044+0000", 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7199999999999998e-07 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.806054+0000", 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.815e-06 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.806077+0000", 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3428e-05 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.3s0 79/78 [PushReplyOp(3:c26b2dda:::smithi09731216-38:head)])", 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:35.830172+0000", 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0036947389999999998, 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040561290000000003, 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.830172+0000", 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.830172+0000", 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.830174+0000", 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.429e-06 2024-04-18T21:52:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.830184+0000", 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0597e-05 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.830208+0000", 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3779999999999999e-05 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.830232+0000", 2024-04-18T21:52:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4142999999999999e-05 2024-04-18T21:52:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:52:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 4 2024-04-18T21:52:35.848 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:35.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:35.897 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:35.896+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:35.967 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:35.967 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:35.967 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:35.967 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:35.967 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:35.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:36.009 INFO:tasks.rados.rados.0.smithi097.stdout:242: finishing write tid 4 to smithi09731216-16 2024-04-18T21:52:36.028 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:36.026+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:36.099 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:36.095+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.847635365999999, 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82320560300000001, 2024-04-18T21:52:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254837+0000", 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9000000000000002e-06 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255465+0000", 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000628435 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255498+0000", 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2305999999999999e-05 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255523+0000", 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5137e-05 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078012+0000", 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82248975599999996 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078039+0000", 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7069000000000001e-05 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.846908460999998, 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82259326700000002, 2024-04-18T21:52:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255563+0000", 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.72e-06 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255574+0000", 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0993e-05 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255598+0000", 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4329999999999999e-05 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255615+0000", 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7569000000000001e-05 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078131+0000", 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82251584300000002 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078154+0000", 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2812000000000001e-05 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:93 3.3s0 3.fef99513 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.846820518999998, 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822627518, 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 93 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255650+0000", 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2330000000000001e-06 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255657+0000", 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5000000000000002e-06 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255681+0000", 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3354e-05 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255705+0000", 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3932999999999999e-05 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078240+0000", 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822535609 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078276+0000", 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5889000000000001e-05 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.117 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:73 3.13s0 3:c8da3643:::smithi09731216-25:head [append 0~409600,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.905737049000003, 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.609707234, 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196734+0000", 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829999999999999e-06 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196977+0000", 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024357200000000001 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196998+0000", 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0503e-05 2024-04-18T21:52:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197013+0000", 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5536000000000002e-05 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197060+0000", 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6519e-05 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197061+0000", 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2900000000000002e-07 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591261+0000", 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3942002449999999 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591644+0000", 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038315000000000002 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593905+0000", 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022609050000000001 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697736+0000", 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10383110700000001 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806236+0000", 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10850008999999999 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806439+0000", 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020309500000000001 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:74 3.13s0 3:c8da3643:::smithi09731216-25:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.905432705000003, 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7102205860000002, 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 74 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197038+0000", 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9000000000000005e-07 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197043+0000", 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197053+0000", 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0267e-05 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197066+0000", 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.327e-05 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197104+0000", 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8039e-05 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197105+0000", 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8400000000000004e-07 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595282+0000", 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3981771869999999 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595543+0000", 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026070899999999997 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.596517+0000", 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00097469500000000003 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697956+0000", 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10143811899999999 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907075+0000", 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20911996799999999 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907257+0000", 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018164799999999999 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.13s0 3:c8a99f7f:::smithi09731216-31:head [append 0~507904,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.904906642, 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.812483013, 2024-04-18T21:52:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8599999999999998e-07 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197879+0000", 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000315495 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197893+0000", 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4143e-05 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197908+0000", 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4932999999999999e-05 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197987+0000", 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9288000000000004e-05 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197989+0000", 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.137e-06 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.698727+0000", 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.500737242 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.699015+0000", 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028849499999999999 2024-04-18T21:52:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.700958+0000", 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019429969999999999 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798717+0000", 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097759288999999999 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.009932+0000", 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.211214651 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010046+0000", 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113657 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.13s0 3:c8a99f7f:::smithi09731216-31:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.904544801, 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8124102710000001, 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197925+0000", 2024-04-18T21:52:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000005e-07 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197931+0000", 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6799999999999998e-06 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197943+0000", 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1620000000000001e-05 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197953+0000", 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-05 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7586000000000001e-05 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5599999999999995e-07 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702431+0000", 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.504419629 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702670+0000", 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023948700000000001 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.703977+0000", 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00130674 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798719+0000", 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094742397000000006 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010219+0000", 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21149920999999999 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010335+0000", 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011602 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:232 3.3s0 3:c170a397:::smithi09731216-3:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.069387433000003, 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.67408736800000002, 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 232 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033082+0000", 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033085+0000", 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.74e-06 2024-04-18T21:52:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033688+0000", 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060374300000000001 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033720+0000", 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2119000000000002e-05 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033737+0000", 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6273000000000002e-05 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.299553+0000", 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.26581622199999999 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.299658+0000", 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105429 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.300593+0000", 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093458899999999999 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.304845+0000", 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042519960000000001 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707032+0000", 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.402187454 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707169+0000", 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013680300000000001 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:233 3.3s0 3:c170a397:::smithi09731216-3:head [setxattr _header (55) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:52:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.068699799000001, 2024-04-18T21:52:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.67365432700000005, 2024-04-18T21:52:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 233 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033769+0000", 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033771+0000", 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2500000000000001e-06 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033776+0000", 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7629999999999997e-06 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033791+0000", 2024-04-18T21:52:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4803e-05 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033801+0000", 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6830000000000003e-06 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.302979+0000", 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.26917756799999998 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.303256+0000", 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027797400000000001 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.304206+0000", 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094934700000000002 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.505420+0000", 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201213902 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707355+0000", 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20193533899999999 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.707424+0000", 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8697999999999999e-05 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:234 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.068652673000003, 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.87684132699999995, 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 234 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9599999999999998e-07 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033821+0000", 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.653e-06 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033834+0000", 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3303e-05 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033847+0000", 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2327000000000001e-05 2024-04-18T21:52:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808105+0000", 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.77425798199999996 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808280+0000", 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017508500000000001 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.910658+0000", 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102378081 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:306 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 38.016136398999997, 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71140329400000002, 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 306 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086333+0000", 2024-04-18T21:52:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086334+0000", 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4900000000000004e-07 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086340+0000", 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0900000000000001e-06 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086359+0000", 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8940000000000002e-05 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086369+0000", 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7569999999999999e-06 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086608+0000", 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000239702 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086634+0000", 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5948999999999998e-05 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.291884+0000", 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20524995300000001 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292051+0000", 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000166797 2024-04-18T21:52:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.797736+0000", 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50568515700000005 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:307 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 38.016086311000002, 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81219510800000005, 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 307 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086384+0000", 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9699999999999995e-07 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086388+0000", 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3900000000000003e-06 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086398+0000", 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8369999999999992e-06 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086406+0000", 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3599999999999996e-06 2024-04-18T21:52:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086727+0000", 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032101900000000002 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086748+0000", 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0917e-05 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292674+0000", 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20592602600000001 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292826+0000", 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015169700000000001 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.898578+0000", 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60575206500000001 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:325 3.3s0 3:c26b2dda:::smithi09731216-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.876295425000002, 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.79501891000000002, 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 325 2024-04-18T21:52:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4400000000000001e-07 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226176+0000", 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1100000000000001e-06 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226182+0000", 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5230000000000003e-06 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226186+0000", 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0180000000000003e-06 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.230252+0000", 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040660369999999998 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.230395+0000", 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014306400000000001 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.817971+0000", 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58757587300000003 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.818220+0000", 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024919800000000001 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:00.021193+0000", 2024-04-18T21:52:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20297274300000001 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:425 3.3s0 3:c022028e:::smithi09731216-15:head [copy-from ver 14] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445446+0000", 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.657022777999998, 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.72780147399999995, 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 425 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445446+0000", 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445446+0000", 2024-04-18T21:52:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445450+0000", 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7129999999999999e-06 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445466+0000", 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.543e-05 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445506+0000", 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0049000000000002e-05 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445531+0000", 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.552e-05 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445699+0000", 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016749399999999999 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445711+0000", 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2616000000000001e-05 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.452009+0000", 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0062980550000000003 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.452196+0000", 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018673999999999999 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.867038+0000", 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41484197699999997 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.873483+0000", 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0064446649999999996 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173090+0000", 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.299607611 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173248+0000", 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015760399999999999 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:432 3.3s0 3:c170a397:::smithi09731216-3:head [copy-from ver 11] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.656757572, 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.728029657, 2024-04-18T21:52:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 432 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445713+0000", 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4759999999999999e-06 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445723+0000", 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.027e-05 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445745+0000", 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.126e-05 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445770+0000", 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5803e-05 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445905+0000", 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013424800000000001 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445914+0000", 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9230000000000008e-06 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.552790+0000", 2024-04-18T21:52:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106876007 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.553298+0000", 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00050816200000000005 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.967658+0000", 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41436041099999998 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.072477+0000", 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104818595 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173605+0000", 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101128601 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173741+0000", 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013590100000000001 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:529 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~786432] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.405381+0000", 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.69708866, 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62624382199999995, 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 529 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.405381+0000", 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.405381+0000", 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.405383+0000", 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5129999999999999e-06 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.406636+0000", 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001253288 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.406667+0000", 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1022999999999998e-05 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.406683+0000", 2024-04-18T21:52:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5985999999999999e-05 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409012+0000", 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002328161 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409201+0000", 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000189827 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.411632+0000", 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024307999999999999 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.519543+0000", 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107911147 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.031484+0000", 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51194032899999997 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.031624+0000", 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000140748 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:530 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~491520] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.408107+0000", 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.694361752000001, 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62388588300000003, 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 530 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408107+0000", 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408107+0000", 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408110+0000", 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6000000000000001e-06 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408800+0000", 2024-04-18T21:52:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069015799999999996 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408832+0000", 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1807000000000002e-05 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.408849+0000", 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6786000000000002e-05 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.513965+0000", 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105116449 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.514105+0000", 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000139877 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.515322+0000", 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012172109999999999 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.520716+0000", 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053939879999999997 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.031865+0000", 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51114865499999995 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.031993+0000", 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012835200000000001 2024-04-18T21:52:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:531 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~278528] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.409700+0000", 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.692769108, 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62260299100000005, 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 531 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409700+0000", 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409700+0000", 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.409702+0000", 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1059999999999998e-06 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410069+0000", 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036714499999999998 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410099+0000", 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9577000000000002e-05 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410117+0000", 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8012999999999998e-05 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.516213+0000", 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106095574 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.516497+0000", 2024-04-18T21:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028478299999999999 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.518393+0000", 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018953959999999999 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.521659+0000", 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032665699999999999 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.032199+0000", 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51053993200000003 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.032303+0000", 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010389499999999999 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:532 3.3s0 3:c2fd059f:::smithi09731216-20:head [setxattr _header (57) in=7b] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.410147+0000", 2024-04-18T21:52:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.692322317, 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.72290332300000004, 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 532 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410147+0000", 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410147+0000", 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410148+0000", 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.083e-06 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410153+0000", 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2460000000000003e-06 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.410167+0000", 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3733999999999999e-05 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510246+0000", 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078801 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.925981+0000", 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41573498199999998 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.926145+0000", 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016407100000000001 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.926979+0000", 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00083374299999999996 2024-04-18T21:52:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.928947+0000", 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001968479 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.132932+0000", 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.203985165 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133050+0000", 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118019 2024-04-18T21:52:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:533 3.3s0 3:c2fd059f:::smithi09731216-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.592203872000001, 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.85709232000000002, 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 533 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510266+0000", 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-06 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510271+0000", 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7670000000000003e-06 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510283+0000", 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1965e-05 2024-04-18T21:52:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.610415+0000", 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100132151 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.927765+0000", 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31734968699999999 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.927918+0000", 2024-04-18T21:52:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000152702 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133621+0000", 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.205703102 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133704+0000", 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3040000000000005e-05 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.367358+0000", 2024-04-18T21:52:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23365385799999999 2024-04-18T21:52:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:36.215 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:36.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:36.217 INFO:tasks.rados.rados.0.smithi097.stdout:246: finishing copy_from racing read to smithi09731216-11 2024-04-18T21:52:36.217 INFO:tasks.rados.rados.0.smithi097.stdout:221: done (7 left) 2024-04-18T21:52:36.217 INFO:tasks.rados.rados.0.smithi097.stdout:240: done (6 left) 2024-04-18T21:52:36.217 INFO:tasks.rados.rados.0.smithi097.stdout:241: done (5 left) 2024-04-18T21:52:36.217 INFO:tasks.rados.rados.0.smithi097.stdout:243: done (4 left) 2024-04-18T21:52:36.217 INFO:tasks.rados.rados.0.smithi097.stdout:247: done (3 left) 2024-04-18T21:52:36.218 INFO:tasks.rados.rados.0.smithi097.stdout:248: delete oid 21 current snap is 19 2024-04-18T21:52:36.218 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:36.219 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:36.210+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:36.259 INFO:tasks.rados.rados.0.smithi097.stdout:246: finishing copy_from to smithi09731216-11 2024-04-18T21:52:36.259 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 11 v 112 (ObjNum 92 snap 18 seq_num 92) dirty exists 2024-04-18T21:52:36.313 INFO:tasks.rados.rados.0.smithi097.stdout:242: finishing write tid 5 to smithi09731216-16 2024-04-18T21:52:36.314 INFO:tasks.rados.rados.0.smithi097.stdout:242: oid 16 updating version 0 to 32 2024-04-18T21:52:36.314 INFO:tasks.rados.rados.0.smithi097.stdout:242: oid 16 version 32 is already newer than 28 2024-04-18T21:52:36.314 INFO:tasks.rados.rados.0.smithi097.stdout:242: oid 16 version 32 is already newer than 29 2024-04-18T21:52:36.314 INFO:tasks.rados.rados.0.smithi097.stdout:242: oid 16 version 32 is already newer than 30 2024-04-18T21:52:36.314 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 16 v 32 (ObjNum 103 snap 19 seq_num 103) dirty exists 2024-04-18T21:52:36.314 INFO:tasks.rados.rados.0.smithi097.stdout:242: left oid 16 (ObjNum 103 snap 19 seq_num 103) 2024-04-18T21:52:36.317 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:36.315+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:36.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:36.380 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:36.395 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:36.395 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:36.396 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:36.396 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:36.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-18T21:52:36.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:36.425 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:36.425 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:36.444 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:36.444 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:36.457 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:36.454+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:36.462 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:36.462 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:36.501 INFO:tasks.rados.rados.0.smithi097.stdout:242: done (3 left) 2024-04-18T21:52:36.501 INFO:tasks.rados.rados.0.smithi097.stdout:246: done (2 left) 2024-04-18T21:52:36.501 INFO:tasks.rados.rados.0.smithi097.stdout:248: done (1 left) 2024-04-18T21:52:36.501 INFO:tasks.rados.rados.0.smithi097.stdout:249: setattr oid 48 current snap is 19 2024-04-18T21:52:36.504 INFO:tasks.rados.rados.0.smithi097.stdout:250: snap_remove snap 4 2024-04-18T21:52:36.519 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:36.516+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:36.536 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:36.536 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:36.536 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:36.536 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:36.536 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:36.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:36.620 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:36.616+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:36.681 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:36.682 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:36.713 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:36.714 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:36.721 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:36.720+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:36.724 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:36.724 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:36.738 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:36.741 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:36.741 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:36.760 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:36.760 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:36.782 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:36.782 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:36.805 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:36.805 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:36.821 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:36.819+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:36.826 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:36.827 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:36.848 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:36.848 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:36.857 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:36.857 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:36.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:36.922 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:36.919+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:36.993 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:36.993 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:37.007 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:37.053 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:37.050+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:37.123 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:37.118+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:37.126 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:37.126 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:37.140 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:37.225 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:37.222+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:37.307 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:37.307 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:37.321 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:37.324 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:37.319+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:37.361 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is unset 2024-04-18T21:52:37.367 INFO:tasks.rados.rados.0.smithi097.stdout:250: done (2 left) 2024-04-18T21:52:37.367 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 11 current snap is 19 2024-04-18T21:52:37.367 INFO:tasks.rados.rados.0.smithi097.stdout:238: finishing rollback tid 1 to smithi09731216-38 2024-04-18T21:52:37.367 INFO:tasks.rados.rados.0.smithi097.stdout:238: finishing rollback tid 2 to smithi09731216-38 2024-04-18T21:52:37.367 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 38 v 131 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-04-18T21:52:37.367 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 48 v 67 (ObjNum 98 snap 19 seq_num 98) dirty exists 2024-04-18T21:52:37.367 INFO:tasks.rados.rados.0.smithi097.stdout:251: seq_num 106 ranges {4374528=622592,4997120=442368,5439488=524288,5963776=196608} 2024-04-18T21:52:37.425 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:37.422+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:37.427 INFO:tasks.rados.rados.0.smithi097.stdout:251: writing smithi09731216-11 from 4374528 to 4997120 tid 1 2024-04-18T21:52:37.431 INFO:tasks.rados.rados.0.smithi097.stdout:251: writing smithi09731216-11 from 4997120 to 5439488 tid 2 2024-04-18T21:52:37.436 INFO:tasks.rados.rados.0.smithi097.stdout:251: writing smithi09731216-11 from 5439488 to 5963776 tid 3 2024-04-18T21:52:37.438 INFO:tasks.rados.rados.0.smithi097.stdout:251: writing smithi09731216-11 from 5963776 to 6160384 tid 4 2024-04-18T21:52:37.438 INFO:tasks.rados.rados.0.smithi097.stdout:238: done (2 left) 2024-04-18T21:52:37.438 INFO:tasks.rados.rados.0.smithi097.stdout:249: done (1 left) 2024-04-18T21:52:37.438 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 20 current snap is 19 2024-04-18T21:52:37.438 INFO:tasks.rados.rados.0.smithi097.stdout:252: seq_num 107 ranges {4374528=638976,5013504=458752,5472256=704512} 2024-04-18T21:52:37.471 INFO:tasks.rados.rados.0.smithi097.stdout:252: writing smithi09731216-20 from 4374528 to 5013504 tid 1 2024-04-18T21:52:37.474 INFO:tasks.rados.rados.0.smithi097.stdout:252: writing smithi09731216-20 from 5013504 to 5472256 tid 2 2024-04-18T21:52:37.479 INFO:tasks.rados.rados.0.smithi097.stdout:252: writing smithi09731216-20 from 5472256 to 6176768 tid 3 2024-04-18T21:52:37.479 INFO:tasks.rados.rados.0.smithi097.stdout:253: copy_from oid 28 from oid 27 current snap is 19 2024-04-18T21:52:37.479 INFO:tasks.rados.rados.0.smithi097.stdout:254: snap_create 2024-04-18T21:52:37.490 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:37.490 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:37.505 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:37.526 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:37.523+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:37.627 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:37.624+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:37.673 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:37.673 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:37.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:37.732 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:37.732+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:37.826 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:37.827 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:37.829 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:37.826+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:37.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:37.930 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:37.927+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:38.008 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:38.008 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:38.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:38.068 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:38.068+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:38.132 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:38.126+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:38.146 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:38.146 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:38.147 INFO:tasks.rados.rados.0.smithi097.stdout:251: finishing write tid 1 to smithi09731216-11 2024-04-18T21:52:38.149 INFO:tasks.rados.rados.0.smithi097.stdout:251: finishing write tid 2 to smithi09731216-11 2024-04-18T21:52:38.150 INFO:tasks.rados.rados.0.smithi097.stdout:251: finishing write tid 3 to smithi09731216-11 2024-04-18T21:52:38.152 INFO:tasks.rados.rados.0.smithi097.stdout:251: finishing write tid 4 to smithi09731216-11 2024-04-18T21:52:38.153 INFO:tasks.rados.rados.0.smithi097.stdout:251: finishing write tid 5 to smithi09731216-11 2024-04-18T21:52:38.160 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:38.211 INFO:tasks.rados.rados.0.smithi097.stdout:253: finishing copy_from racing read to smithi09731216-28 2024-04-18T21:52:38.248 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:38.248+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:38.327 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:38.327 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:38.334 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:38.331+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:38.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:38.369 INFO:tasks.rados.rados.0.smithi097.stdout:254: done (3 left) 2024-04-18T21:52:38.369 INFO:tasks.rados.rados.0.smithi097.stdout:255: snap_create 2024-04-18T21:52:38.398 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [0, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:38.398 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:52:38.398 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-04-18T21:52:38.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 11 2024-04-18T21:52:38.417 INFO:tasks.rados.rados.0.smithi097.stdout:253: finishing copy_from to smithi09731216-28 2024-04-18T21:52:38.417 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 28 v 13 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-18T21:52:38.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:38.431+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:38.463 INFO:tasks.rados.rados.0.smithi097.stdout:251: finishing write tid 6 to smithi09731216-11 2024-04-18T21:52:38.463 INFO:tasks.rados.rados.0.smithi097.stdout:251: oid 11 updating version 0 to 116 2024-04-18T21:52:38.463 INFO:tasks.rados.rados.0.smithi097.stdout:251: oid 11 version 116 is already newer than 115 2024-04-18T21:52:38.463 INFO:tasks.rados.rados.0.smithi097.stdout:251: oid 11 version 116 is already newer than 113 2024-04-18T21:52:38.463 INFO:tasks.rados.rados.0.smithi097.stdout:251: oid 11 version 116 is already newer than 114 2024-04-18T21:52:38.463 INFO:tasks.rados.rados.0.smithi097.stdout:251: oid 11 updating version 116 to 117 2024-04-18T21:52:38.464 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 11 v 117 (ObjNum 106 snap 19 seq_num 106) dirty exists 2024-04-18T21:52:38.464 INFO:tasks.rados.rados.0.smithi097.stdout:251: left oid 11 (ObjNum 106 snap 19 seq_num 106) 2024-04-18T21:52:38.508 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:38.508 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:38.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:38.535 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:38.530+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:38.637 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:38.636+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:38.647 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:38.647 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:38.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:38.728 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-18T21:52:38.723+0000 7f0412e7b640 -1 mon.a@0(leader).osd e82 definitely_dead 0 2024-04-18T21:52:38.737 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:38.735+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:38.830 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:38.831 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:38.838 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:38.836+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:38.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:38.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:38.939 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:38.934+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:39.014 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:39.014 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:39.021 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:39.021 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:39.028 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:39.040 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:39.035+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:39.105 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:39.106 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:39.132 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:39.132 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:39.140 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:39.135+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:39.152 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:39.153 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:39.156 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:39.156 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:39.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:39.182 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:39.182 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:39.241 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:39.239+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:39.251 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:39.251 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:39.283 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:39.283 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:39.290 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:39.291 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:39.305 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:39.318 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:39.318 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:39.342 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:39.339+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:39.342 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:39.343 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:39.368 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:39.369 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:39.370 INFO:teuthology.orchestra.run.smithi097.stderr:marked in osd.11. 2024-04-18T21:52:39.373 INFO:tasks.rados.rados.0.smithi097.stdout:251: done (3 left) 2024-04-18T21:52:39.373 INFO:tasks.rados.rados.0.smithi097.stdout:253: done (2 left) 2024-04-18T21:52:39.373 INFO:tasks.rados.rados.0.smithi097.stdout:255: done (1 left) 2024-04-18T21:52:39.373 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 50 current snap is 21 2024-04-18T21:52:39.373 INFO:tasks.rados.rados.0.smithi097.stdout:256: seq_num 108 ranges {32768=655360,688128=606208,1294336=475136,1769472=81920} 2024-04-18T21:52:39.378 INFO:tasks.rados.rados.0.smithi097.stdout:256: writing smithi09731216-50 from 32768 to 688128 tid 1 2024-04-18T21:52:39.384 INFO:tasks.rados.rados.0.smithi097.stdout:256: writing smithi09731216-50 from 688128 to 1294336 tid 2 2024-04-18T21:52:39.389 INFO:tasks.rados.rados.0.smithi097.stdout:256: writing smithi09731216-50 from 1294336 to 1769472 tid 3 2024-04-18T21:52:39.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-18T21:52:39.391 INFO:tasks.rados.rados.0.smithi097.stdout:256: writing smithi09731216-50 from 1769472 to 1851392 tid 4 2024-04-18T21:52:39.392 INFO:tasks.rados.rados.0.smithi097.stdout:257: rollback oid 5 current snap is 21 2024-04-18T21:52:39.392 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 5 to 14 2024-04-18T21:52:39.392 INFO:tasks.rados.rados.0.smithi097.stdout:258: rmattr oid 3 current snap is 21 2024-04-18T21:52:39.392 INFO:tasks.rados.rados.0.smithi097.stdout:258: done (3 left) 2024-04-18T21:52:39.392 INFO:tasks.rados.rados.0.smithi097.stdout:259: delete oid 19 current snap is 21 2024-04-18T21:52:39.401 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-04-18T21:52:39.401 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:39.401 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:39.422 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:39.422 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:39.436 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:39.437 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:39.442 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:39.438+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:39.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:39.543 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:39.539+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:106 3.6s0 3:64491ca4:::smithi09731216-36:head [append 0~589824,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.485547967999999, 2024-04-18T21:52:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.39237912, 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 106 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088750+0000", 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3859999999999999e-06 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089602+0000", 2024-04-18T21:52:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000852537 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089629+0000", 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6066000000000001e-05 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089648+0000", 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9449999999999998e-05 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174464+0000", 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816027000000002 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174469+0000", 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5970000000000002e-06 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481023+0000", 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3065548339999999 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481128+0000", 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010422299999999999 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:107 3.6s0 3:64491ca4:::smithi09731216-36:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.484619004999999, 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.301432395, 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 107 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089679+0000", 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.686e-06 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089697+0000", 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7717e-05 2024-04-18T21:52:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089714+0000", 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7433000000000001e-05 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089730+0000", 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5953000000000001e-05 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174547+0000", 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816707000000005 2024-04-18T21:52:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174550+0000", 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9100000000000001e-06 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.983958+0000", 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8094078099999999 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984087+0000", 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012884800000000001 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984411+0000", 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032455500000000003 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986825+0000", 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024135609999999998 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391001+0000", 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40417611399999998 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391110+0000", 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109101 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:108 3.6s0 3:64491ca4:::smithi09731216-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.484545359999998, 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4199299170000002, 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 108 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089753+0000", 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.46e-06 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089759+0000", 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5969999999999997e-06 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089782+0000", 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2546999999999999e-05 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089799+0000", 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6742999999999999e-05 2024-04-18T21:52:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174653+0000", 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084854180000000001 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.886e-06 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391631+0000", 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2169758150000001 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391728+0000", 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6861000000000007e-05 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509681+0000", 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.117952828 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:109 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~606208,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.484471806999998, 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4179864210000002, 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109 2024-04-18T21:52:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089826+0000", 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.147e-06 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090694+0000", 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00086783200000000002 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090726+0000", 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2379999999999998e-05 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090746+0000", 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9466999999999998e-05 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174599+0000", 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083853091000000005 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174600+0000", 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.097e-06 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985378+0000", 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.810778236 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985469+0000", 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0805999999999998e-05 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986312+0000", 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084315300000000002 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089266+0000", 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10295370099999999 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507611+0000", 2024-04-18T21:52:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41834516199999999 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507811+0000", 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000200349 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:110 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.483528403999998, 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.429779607, 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 110 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090769+0000", 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090782+0000", 2024-04-18T21:52:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2547e-05 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090797+0000", 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5109000000000001e-05 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090807+0000", 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8670000000000006e-06 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083849218000000003 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174657+0000", 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8700000000000004e-07 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087432+0000", 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9127752569999998 2024-04-18T21:52:39.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087519+0000", 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6668000000000004e-05 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088034+0000", 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00051538599999999995 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089552+0000", 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001518295 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520465+0000", 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.43091233000000001 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520548+0000", 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2947000000000004e-05 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:111 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.483478665, 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5330603059999999, 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 111 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090819+0000", 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6000000000000003e-07 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090822+0000", 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.066e-06 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090834+0000", 2024-04-18T21:52:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2486999999999999e-05 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090844+0000", 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9899999999999992e-06 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275049+0000", 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.184205065 2024-04-18T21:52:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275055+0000", 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4389999999999999e-06 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088362+0000", 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8133068830000001 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088621+0000", 2024-04-18T21:52:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025974499999999997 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520968+0000", 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.432346972 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521091+0000", 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122263 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.623878+0000", 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102787636 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:127 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~720896,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.760003679999997, 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50504046999999996, 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 127 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814297+0000", 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3120000000000004e-06 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815282+0000", 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0009845469999999999 2024-04-18T21:52:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815306+0000", 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4488999999999999e-05 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815321+0000", 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4365999999999999e-05 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901001+0000", 2024-04-18T21:52:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085680191000000003 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901126+0000", 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125506 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902346+0000", 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012201989999999999 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904676+0000", 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023299459999999998 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319226+0000", 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.414549324 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319333+0000", 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010759 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:128 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.758948924000002, 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70473296200000002, 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 128 2024-04-18T21:52:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815349+0000", 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.003e-06 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815353+0000", 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5700000000000003e-06 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815362+0000", 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4540000000000007e-06 2024-04-18T21:52:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815369+0000", 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4059999999999999e-06 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903371+0000", 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088001653999999999 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903511+0000", 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000139889 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904181+0000", 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067014400000000001 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.905274+0000", 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010929430000000001 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519876+0000", 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61460184500000004 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520081+0000", 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020505400000000001 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.1s0,GRANT,e:32,from:5(4),reservation_nonce:5)", 2024-04-18T21:52:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.672762673999998, 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61973115499999998, 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901535+0000", 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.206e-06 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901541+0000", 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.331e-06 2024-04-18T21:52:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901559+0000", 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7977000000000001e-05 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.001690+0000", 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100130964 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521089+0000", 2024-04-18T21:52:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51939875099999999 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521142+0000", 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3324e-05 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521265+0000", 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122602 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:130 3.1s0 3:87c089d9:::smithi09731216-44:head [append 0~737280,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.663290404999998, 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61075046099999997, 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 130 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911008+0000", 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.356e-06 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911859+0000", 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085008399999999997 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911877+0000", 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8147000000000001e-05 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911886+0000", 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012129+0000", 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100243058 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012391+0000", 2024-04-18T21:52:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026173100000000002 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.014631+0000", 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002240344 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118257+0000", 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103625859 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521564+0000", 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40330724200000001 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521757+0000", 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019236000000000001 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:131 3.1s0 3:87c089d9:::smithi09731216-44:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.662392095999998, 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70851564899999997, 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 131 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911905+0000", 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9800000000000002e-07 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911910+0000", 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4499999999999997e-06 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911919+0000", 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911927+0000", 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5700000000000004e-06 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017049+0000", 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105122775 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017158+0000", 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010845600000000001 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017640+0000", 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00048220899999999999 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118261+0000", 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100621063 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620272+0000", 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50201088400000005 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620420+0000", 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000148064 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.467407653999999, 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31260053300000001, 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106890+0000", 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8599999999999996e-07 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106896+0000", 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9139999999999996e-06 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106911+0000", 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4800999999999999e-05 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106920+0000", 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9379999999999998e-06 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419470+0000", 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.312550204 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419489+0000", 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.969e-05 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.369482881000003, 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.316195273, 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1699999999999998e-07 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204818+0000", 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5659999999999998e-06 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204828+0000", 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0478e-05 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204836+0000", 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.413e-06 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519654+0000", 2024-04-18T21:52:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.314818722 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521009+0000", 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001354577 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=13, last_complete=32'5, committed=1, applied=1))", 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.369449195999998, 2024-04-18T21:52:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31487986099999998, 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204848+0000", 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8299999999999998e-07 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204850+0000", 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6429999999999999e-06 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204859+0000", 2024-04-18T21:52:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0550000000000005e-06 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204866+0000", 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5320000000000003e-06 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519669+0000", 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31480259199999999 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519727+0000", 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8656000000000002e-05 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.369416338999997, 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31508690700000003, 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204881+0000", 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3000000000000002e-07 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204884+0000", 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.976e-06 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204900+0000", 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6418000000000001e-05 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204908+0000", 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3080000000000008e-06 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519923+0000", 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31501511199999999 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519967+0000", 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3763000000000001e-05 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:524 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~770048] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.380459+0000", 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.193837954999999, 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.709551441, 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.653 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 524 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.380459+0000", 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.380459+0000", 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.380463+0000", 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0130000000000004e-06 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382388+0000", 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019258980000000001 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382426+0000", 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7521000000000002e-05 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382449+0000", 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3371999999999999e-05 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386170+0000", 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037207609999999999 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386450+0000", 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027958300000000002 2024-04-18T21:52:39.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.390115+0000", 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036657510000000001 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.403183+0000", 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.013067066 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.089832+0000", 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.68664989300000001 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.090010+0000", 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017758300000000001 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:525 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~475136] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.385002+0000", 2024-04-18T21:52:39.658 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.189294969000001, 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.80581454200000002, 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 525 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.385002+0000", 2024-04-18T21:52:39.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.385002+0000", 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.385004+0000", 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8660000000000002e-06 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386191+0000", 2024-04-18T21:52:39.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001186335 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386225+0000", 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4538e-05 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386248+0000", 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2402e-05 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.395454+0000", 2024-04-18T21:52:39.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0092064380000000008 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.395619+0000", 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000165274 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.397339+0000", 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017190899999999999 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.404292+0000", 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069534590000000004 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.190633+0000", 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78634107099999995 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.190816+0000", 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018306900000000001 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.663 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:526 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~294912] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.386285+0000", 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.188011406999999, 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.80528929800000004, 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 526 2024-04-18T21:52:39.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386285+0000", 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386285+0000", 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386288+0000", 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6240000000000002e-06 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386930+0000", 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00064243500000000001 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386966+0000", 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5340999999999999e-05 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386994+0000", 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8289e-05 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.398679+0000", 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01168482 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.398836+0000", 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000157355 2024-04-18T21:52:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.400287+0000", 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014512450000000001 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.404323+0000", 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040361559999999999 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.191308+0000", 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78698441100000005 2024-04-18T21:52:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.191574+0000", 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000266622 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:527 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [setxattr _header (57) in=7b] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.387028+0000", 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.187268496, 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90512298099999999, 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 527 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387028+0000", 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387028+0000", 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387030+0000", 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.866e-06 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387039+0000", 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9060000000000008e-06 2024-04-18T21:52:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387060+0000", 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0979e-05 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387076+0000", 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6175999999999999e-05 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.401384+0000", 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.014307998000000001 2024-04-18T21:52:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.401528+0000", 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014453400000000001 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.402440+0000", 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00091111999999999996 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.404362+0000", 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001922656 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.291910+0000", 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88754813200000005 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.292151+0000", 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000240614 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:528 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.387098+0000", 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.187198317, 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2983067880000001, 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 528 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387098+0000", 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387098+0000", 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387099+0000", 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.243e-06 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387106+0000", 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0589999999999999e-06 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387125+0000", 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9488999999999999e-05 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387159+0000", 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4372000000000002e-05 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.402967+0000", 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015808106999999998 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.403155+0000", 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018745599999999999 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.293393+0000", 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.89023824600000001 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.293571+0000", 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017742999999999999 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.685405+0000", 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39183438599999998 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:39.679 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:39.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:39.681 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:39.678+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:39.698 INFO:tasks.rados.rados.0.smithi097.stdout:257: finishing rollback tid 0 to smithi09731216-5 2024-04-18T21:52:39.699 INFO:tasks.rados.rados.0.smithi097.stdout:257: finishing rollback tid 1 to smithi09731216-5 2024-04-18T21:52:39.782 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:39.778+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:39.798 INFO:tasks.rados.rados.0.smithi097.stdout:257: finishing rollback tid 2 to smithi09731216-5 2024-04-18T21:52:39.799 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 5 v 30 (ObjNum 77 snap 10 seq_num 77) dirty exists 2024-04-18T21:52:39.805 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:39.806 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:39.806 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:39.806 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:39.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:39.905 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:39.903+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:39.985 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:39.984+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:39.986 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:39.986 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:39.986 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:39.986 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:39.986 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:39.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:40.085 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:40.083+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:40.164 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:155 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'109, mlcod=24'105)", 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.856780542000003, 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30724263200000002, 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6800000000000002e-07 2024-04-18T21:52:40.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306525+0000", 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2228999999999999e-05 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306532+0000", 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7469999999999999e-06 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306539+0000", 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1250000000000004e-06 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.307270+0000", 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00073067900000000003 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.307309+0000", 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9830000000000003e-05 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613662+0000", 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30635288799999999 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613735+0000", 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2866000000000003e-05 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.169 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:156 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'110, mlcod=24'105)", 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.856727769000003, 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30725738200000002, 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306546+0000", 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.16e-07 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306548+0000", 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.672e-06 2024-04-18T21:52:40.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306553+0000", 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0730000000000004e-06 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306556+0000", 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.959e-06 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408193+0000", 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101636647 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408269+0000", 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5767999999999997e-05 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613742+0000", 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.205473081 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613803+0000", 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0965999999999999e-05 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:157 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'111, mlcod=24'105)", 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.856712805999997, 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30730748800000002, 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.174 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.72e-07 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306563+0000", 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2730000000000001e-06 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306572+0000", 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2390000000000008e-06 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306577+0000", 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7709999999999997e-06 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408635+0000", 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102058725 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408654+0000", 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8921000000000001e-05 2024-04-18T21:52:40.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613808+0000", 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20515377100000001 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613868+0000", 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9616e-05 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.800796059, 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40570043900000002, 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362478+0000", 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4279999999999999e-06 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362656+0000", 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017710100000000001 2024-04-18T21:52:40.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362673+0000", 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7802e-05 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462745+0000", 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100071484 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.767971+0000", 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30522610500000003 2024-04-18T21:52:40.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.768177+0000", 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020651900000000001 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.733906822999998, 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44082012100000001, 2024-04-18T21:52:40.182 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429367+0000", 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5300000000000004e-07 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429643+0000", 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027611500000000001 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429665+0000", 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1603e-05 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.529793+0000", 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100128809 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868616+0000", 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.33882269300000001 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870186+0000", 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015703480000000001 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.185 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.700436951999997, 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40574496100000002, 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.186 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0100000000000001e-07 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462933+0000", 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6247000000000002e-05 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462943+0000", 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.944e-06 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462948+0000", 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4979999999999997e-06 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868491+0000", 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40554244699999997 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868581+0000", 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0524000000000003e-05 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.697526547000002, 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71932642099999999, 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:40.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9599999999999998e-07 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465774+0000", 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6846e-05 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465781+0000", 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2640000000000002e-06 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465785+0000", 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7759999999999999e-06 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.185006+0000", 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71922088500000003 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.185073+0000", 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6853999999999994e-05 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.697457534999998, 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82169303100000002, 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.193 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4000000000000002e-07 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465848+0000", 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1609000000000002e-05 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465853+0000", 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.366e-06 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465856+0000", 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.151e-06 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286522+0000", 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82066616299999995 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.287509+0000", 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0009863020000000001 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.69736073, 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40453295099999997, 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465913+0000", 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6199999999999998e-07 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465996+0000", 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3168000000000001e-05 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.466007+0000", 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0662000000000001e-05 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.466014+0000", 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8639999999999998e-06 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870275+0000", 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.404261702 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870445+0000", 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016999300000000001 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:40.200 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.631892630999999, 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.75509254699999995, 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:40.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531387+0000", 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2899999999999999e-06 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531557+0000", 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016995899999999999 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531588+0000", 2024-04-18T21:52:40.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1069999999999999e-05 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531606+0000", 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8479999999999999e-05 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286385+0000", 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.754778964 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286473+0000", 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7783999999999998e-05 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.597183004000001, 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858464, 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.204 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566092+0000", 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5799999999999999e-06 2024-04-18T21:52:40.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566247+0000", 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155613 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566277+0000", 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0111999999999999e-05 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566315+0000", 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7141999999999998e-05 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.974376+0000", 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.408061795 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.974675+0000", 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029839800000000001 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=7, attrs_read=0))", 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.156506071000003, 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41262958999999999, 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.208 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006768+0000", 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4139999999999999e-06 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006798+0000", 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9499e-05 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006823+0000", 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4706999999999999e-05 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006842+0000", 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9507000000000001e-05 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418445+0000", 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41160240300000001 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419397+0000", 2024-04-18T21:52:40.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095206000000000002 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.10s1 41/39 ECSubRead(tid=11, to_read={3:0e754656:::smithi09731216-19:head=0,4096,0}, subchunks={3:0e754656:::smithi09731216-19:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.141682467000003, 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39881851499999998, 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021616+0000", 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5575000000000002e-05 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021627+0000", 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1099e-05 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.420045+0000", 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39841796699999998 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.420409+0000", 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036387399999999998 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.141031478000002, 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39759610699999998, 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:40.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6199999999999997e-07 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022253+0000", 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0283e-05 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022265+0000", 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2302e-05 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022271+0000", 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2550000000000003e-06 2024-04-18T21:52:40.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419761+0000", 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39748950500000002 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419838+0000", 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7000000000000001e-05 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.140807705999997, 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39718676600000002, 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022467+0000", 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3030000000000001e-06 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022485+0000", 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8363999999999999e-05 2024-04-18T21:52:40.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022504+0000", 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9283000000000002e-05 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022514+0000", 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8430000000000005e-06 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419513+0000", 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39699880100000001 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419652+0000", 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013917199999999999 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:40.223 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.140688830000002, 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39738200800000001, 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022585+0000", 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7599999999999996e-07 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022601+0000", 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6203000000000001e-05 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022613+0000", 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1334e-05 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022618+0000", 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4110000000000004e-06 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419896+0000", 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39727772700000002 2024-04-18T21:52:40.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419966+0000", 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0556999999999995e-05 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.10s1 41/39 ECSubWriteReply(tid=10, last_complete=41'16, committed=1, applied=1))", 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.140636233999999, 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39534124999999998, 2024-04-18T21:52:40.227 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022640+0000", 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0029999999999999e-06 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022652+0000", 2024-04-18T21:52:40.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1585e-05 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022656+0000", 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5739999999999997e-06 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.417932+0000", 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39527527800000001 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.417978+0000", 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6350000000000002e-05 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.10s1 41/39 ECSubWriteReply(tid=10, last_complete=41'16, committed=1, applied=1))", 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.058575935999997, 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31366916900000003, 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.231 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104699+0000", 2024-04-18T21:52:40.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.686e-06 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104713+0000", 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4296e-05 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104742+0000", 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9204999999999999e-05 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104758+0000", 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5254e-05 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418210+0000", 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31345283600000001 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418366+0000", 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155892 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.234 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 3.18s6,from:0'0,to:38'13,epoch:41/31,start:3:18000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.057147411999999, 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31261068600000003, 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106127+0000", 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5000000000000001e-07 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106133+0000", 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8940000000000004e-06 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106149+0000", 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5534000000000001e-05 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106159+0000", 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.047e-05 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418620+0000", 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31246021499999999 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418658+0000", 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8566000000000003e-05 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418736+0000", 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8157000000000004e-05 2024-04-18T21:52:40.238 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.041582308999999, 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39824586299999998, 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121692+0000", 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5499999999999997e-07 2024-04-18T21:52:40.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121705+0000", 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3207000000000001e-05 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121718+0000", 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2913e-05 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121724+0000", 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5030000000000002e-06 2024-04-18T21:52:40.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.519849+0000", 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39812503900000001 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.519937+0000", 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7346e-05 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.242 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.243 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:40.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:40.244 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:40.240+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:40.345 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:40.344+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:40.377 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is set 2024-04-18T21:52:40.421 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:40.421 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:40.421 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:40.421 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:40.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:40.445 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:40.442+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:40.546 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:40.544+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:40.603 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:40.603 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:40.603 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:40.603 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:40.604 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:40.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:40.647 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:40.644+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:40.743 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:40.743 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:40.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:40.743 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:94 3.0s0 3:033c99c3:::smithi09731216-32:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 54.186381034999997, 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40893576500000001, 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94 2024-04-18T21:52:40.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555853+0000", 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1889999999999999e-06 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556617+0000", 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076362200000000004 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556649+0000", 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2478999999999999e-05 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556665+0000", 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5837000000000001e-05 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556826+0000", 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016137200000000001 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.557241+0000", 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00041440699999999999 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.559642+0000", 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024011430000000001 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563415+0000", 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003773233 2024-04-18T21:52:40.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964577+0000", 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40116224099999997 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964786+0000", 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000208242 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:95 3.0s0 3:033c99c3:::smithi09731216-32:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 54.185527938, 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858905099999998, 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 95 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.750 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556705+0000", 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6360000000000001e-06 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556715+0000", 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.066e-05 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556734+0000", 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8323999999999999e-05 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556745+0000", 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1240000000000001e-05 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561327+0000", 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0045823330000000001 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561583+0000", 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000255732 2024-04-18T21:52:40.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.562607+0000", 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010240990000000001 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563867+0000", 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012605439999999999 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965078+0000", 2024-04-18T21:52:40.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40121052699999998 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965292+0000", 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021395600000000001 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:96 3.0s0 3:033c99c3:::smithi09731216-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:40.755 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 54.185430396000001, 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42323290800000002, 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 96 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:40.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556802+0000", 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7400000000000001e-06 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556808+0000", 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.993e-06 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556828+0000", 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9364e-05 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.656965+0000", 2024-04-18T21:52:40.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10013702300000001 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762039+0000", 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105074903 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762350+0000", 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031089399999999998 2024-04-18T21:52:40.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967780+0000", 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20542997700000001 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.972542+0000", 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004761932 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.979990+0000", 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0074475289999999996 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.980033+0000", 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3553000000000001e-05 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:222 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:40.761 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.729348807000001, 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30521668800000001, 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 222 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012882+0000", 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012885+0000", 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7760000000000002e-06 2024-04-18T21:52:40.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012901+0000", 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6563e-05 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012934+0000", 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2719999999999998e-05 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.012952+0000", 2024-04-18T21:52:40.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7730000000000001e-05 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013179+0000", 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022670500000000001 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013200+0000", 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1466000000000001e-05 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114024+0000", 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100823616 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114179+0000", 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015544500000000001 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.318099+0000", 2024-04-18T21:52:40.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203919667 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:223 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.729228378000002, 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40441434900000001, 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.767 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 223 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013002+0000", 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013004+0000", 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5400000000000001e-06 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013015+0000", 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0519999999999999e-05 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013077+0000", 2024-04-18T21:52:40.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2848999999999994e-05 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013098+0000", 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1083000000000001e-05 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013359+0000", 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000260408 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013383+0000", 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3867e-05 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114847+0000", 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10146374399999999 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.114996+0000", 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149865 2024-04-18T21:52:40.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.417417+0000", 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30242047300000002 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:224 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.729098286000003, 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40564824999999999, 2024-04-18T21:52:40.772 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 224 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013134+0000", 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.387e-06 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013144+0000", 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0273e-05 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013166+0000", 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1857000000000001e-05 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013182+0000", 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5645999999999999e-05 2024-04-18T21:52:40.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113542+0000", 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10035996599999999 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113552+0000", 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0492999999999999e-05 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215576+0000", 2024-04-18T21:52:40.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102023869 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215730+0000", 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000153635 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.418781+0000", 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203051124 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:302 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.680579965, 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.406305586, 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 302 2024-04-18T21:52:40.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061654+0000", 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061669+0000", 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4637000000000001e-05 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061699+0000", 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9791e-05 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061714+0000", 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5441999999999999e-05 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061811+0000", 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7153999999999998e-05 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061819+0000", 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9279999999999993e-06 2024-04-18T21:52:40.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064181+0000", 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002361447 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064306+0000", 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012514399999999999 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.782 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.065234+0000", 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092848599999999996 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.165102+0000", 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.099867808000000002 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467877+0000", 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.302775085 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467956+0000", 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9381000000000005e-05 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:303 3.0s0 3:00e42e84:::smithi09731216-41:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.680480574999997, 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50679023400000001, 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.784 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 303 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061751+0000", 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5499999999999998e-07 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061754+0000", 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2339999999999999e-06 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061763+0000", 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3950000000000007e-06 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061769+0000", 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9730000000000002e-06 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061904+0000", 2024-04-18T21:52:40.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013470999999999999 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061911+0000", 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2200000000000003e-06 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163859+0000", 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101947416 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163977+0000", 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117971 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.164692+0000", 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071513499999999999 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267070+0000", 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102378067 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568454+0000", 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.301384333 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568541+0000", 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6224999999999995e-05 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:304 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.680438103, 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60693648, 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 304 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:52:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3000000000000001e-07 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061797+0000", 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3680000000000002e-06 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061804+0000", 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0809999999999996e-06 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061809+0000", 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5280000000000003e-06 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061950+0000", 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000140529 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061958+0000", 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5329999999999994e-06 2024-04-18T21:52:40.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166016+0000", 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104057925 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166118+0000", 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010206500000000001 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166690+0000", 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057174600000000004 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267073+0000", 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100382727 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668597+0000", 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401524149 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668729+0000", 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132399 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:332 3.0s0 3:00b6e030:::smithi09731216-21:head [copy-from ver 39] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.536279124000004, 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.72786893100000005, 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.797 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 332 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205954+0000", 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4370000000000001e-06 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205969+0000", 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4516e-05 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206001+0000", 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.252e-05 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206022+0000", 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.084e-05 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306220+0000", 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100197961 2024-04-18T21:52:40.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306483+0000", 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026249200000000002 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.726594+0000", 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42011143499999998 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.730261+0000", 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036666060000000002 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933714+0000", 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203453467 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933821+0000", 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010665700000000001 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.802 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:333 3.0s0 3:00b6e030:::smithi09731216-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.536174150999997, 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52393701999999998, 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.803 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 333 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206058+0000", 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.426e-06 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206065+0000", 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4799999999999998e-06 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206085+0000", 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0089999999999999e-05 2024-04-18T21:52:40.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206101+0000", 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.643e-05 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306839+0000", 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100738203 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:52:40.806 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306987+0000", 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014755599999999999 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729615+0000", 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42262781100000002 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729821+0000", 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020588799999999999 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729994+0000", 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000173136 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=46, reqid=client.4509.0:360, at_version=47'10, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.917691380999997, 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40258300600000002, 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.808 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824539+0000", 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824542+0000", 2024-04-18T21:52:40.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.193e-06 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824777+0000", 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234878 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824808+0000", 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1633e-05 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924923+0000", 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10011492700000001 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.025179+0000", 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100256004 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.025197+0000", 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8326e-05 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227086+0000", 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201888453 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227122+0000", 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6591999999999998e-05 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=47, reqid=client.4509.0:365, at_version=47'11, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.817264647000002, 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218968200000002, 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924966+0000", 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.924967+0000", 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.088e-06 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925088+0000", 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120229 2024-04-18T21:52:40.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925104+0000", 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6804e-05 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925115+0000", 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0516999999999999e-05 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.026507+0000", 2024-04-18T21:52:40.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101391776 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.026542+0000", 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5126999999999998e-05 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227138+0000", 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20059612399999999 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227156+0000", 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8017e-05 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=48, reqid=client.4509.0:366, at_version=47'12, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.817097007999998, 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30205703299999997, 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.817 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925134+0000", 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2500000000000001e-07 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925294+0000", 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016000000000000001 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925308+0000", 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4144999999999999e-05 2024-04-18T21:52:40.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925319+0000", 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.008e-05 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.027610+0000", 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102291967 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.820 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.027638+0000", 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7453e-05 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227173+0000", 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19953516700000001 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227191+0000", 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7796e-05 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.821 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=49, reqid=client.4509.0:367, at_version=47'13, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.816899225999997, 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30187523100000002, 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925332+0000", 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4700000000000002e-07 2024-04-18T21:52:40.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925351+0000", 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8998e-05 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925362+0000", 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1302e-05 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925370+0000", 2024-04-18T21:52:40.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6680000000000001e-06 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.028840+0000", 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10347008100000001 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.028867+0000", 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6982999999999999e-05 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227194+0000", 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19832709500000001 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227207+0000", 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2656999999999999e-05 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.11s4 47/45 ECSubWrite(tid=50, reqid=client.4509.0:361, at_version=47'14, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:40.826 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 36.816849030999997, 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30184082099999998, 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925382+0000", 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6600000000000002e-07 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925457+0000", 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4703000000000006e-05 2024-04-18T21:52:40.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925469+0000", 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1932e-05 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.925477+0000", 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1529999999999997e-06 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.029353+0000", 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10387573999999999 2024-04-18T21:52:40.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.029380+0000", 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7033e-05 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227210+0000", 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19783015700000001 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.830 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.227223+0000", 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2737e-05 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:430 3.0s0 3:00b6e030:::smithi09731216-21:head [copy-from ver 39] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.296504631000001, 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71883465800000002, 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.831 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 430 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445730+0000", 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7629999999999998e-06 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445749+0000", 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8960000000000001e-05 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445786+0000", 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7049999999999999e-05 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445804+0000", 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8236e-05 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445984+0000", 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017996200000000001 2024-04-18T21:52:40.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446012+0000", 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7554000000000001e-05 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.515978+0000", 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069965816 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516184+0000", 2024-04-18T21:52:40.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020612199999999999 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.970820+0000", 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45463621900000001 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.978038+0000", 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0072179430000000001 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.164404+0000", 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18636640400000001 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.164561+0000", 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015662900000000001 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.837 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:431 3.0s0 3:00b6e030:::smithi09731216-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 26.296378878999999, 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52894810699999995, 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 431 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445853+0000", 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.437e-06 2024-04-18T21:52:40.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445863+0000", 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4769999999999992e-06 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445885+0000", 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2093e-05 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445901+0000", 2024-04-18T21:52:40.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5826999999999999e-05 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446183+0000", 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028208399999999999 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446201+0000", 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8133000000000001e-05 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516497+0000", 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.070296147000000003 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516639+0000", 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000141533 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974265+0000", 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45762602699999999 2024-04-18T21:52:40.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974495+0000", 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022996099999999999 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974800+0000", 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030538800000000002 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.843 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:560 3.0s0 3:00e42e84:::smithi09731216-41:head [copy-from ver 4] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.723804406999999, 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41315372, 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 560 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.844 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018429+0000", 2024-04-18T21:52:40.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.357e-06 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018443+0000", 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4491999999999999e-05 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018468+0000", 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5060000000000001e-05 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018488+0000", 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9259e-05 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.118707+0000", 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10021904099999999 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.118865+0000", 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015874199999999999 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.226410+0000", 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107544398 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.229100+0000", 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026898130000000001 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.431493+0000", 2024-04-18T21:52:40.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20239311800000001 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.431580+0000", 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7440000000000003e-05 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 73/31 ECSubWriteReply(tid=52, last_complete=73'30, committed=1, applied=1))", 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.534395+0000", 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.207835499, 2024-04-18T21:52:40.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30324524800000002, 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534395+0000", 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534395+0000", 2024-04-18T21:52:40.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534396+0000", 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.02e-07 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534402+0000", 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6799999999999998e-06 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:40.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.534416+0000", 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3848e-05 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.634508+0000", 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10009191000000001 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.837557+0000", 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20304905200000001 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:40.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:40.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.837641+0000", 2024-04-18T21:52:40.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3856000000000005e-05 2024-04-18T21:52:40.853 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.853 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.853 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.853 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:40.853 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:40.853 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:40.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:40.855 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:40.850+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:40.956 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:40.951+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:40.980 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:40.981 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:40.981 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:40.981 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:40.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:41.081 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:41.079+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:41.158 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:41.155+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:41.158 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:41.158 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:41.158 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:41.159 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:41.159 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:41.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:41.259 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:41.254+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:41.338 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:41.338 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:41.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:41.338 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:41.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.338 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:83 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:41.338 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:41.338 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 55.284981821999999, 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28825372900000001, 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 83 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050825+0000", 2024-04-18T21:52:41.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2e-07 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050828+0000", 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3929999999999998e-06 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050832+0000", 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4909999999999999e-06 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050836+0000", 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5219999999999999e-06 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052455+0000", 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016189710000000001 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052552+0000", 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7213999999999995e-05 2024-04-18T21:52:41.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.052983+0000", 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043119300000000002 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138342+0000", 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.085359090999999998 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.338942+0000", 2024-04-18T21:52:41.343 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.200600308 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339078+0000", 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136026 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:84 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:41.344 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 55.284953164000001, 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59192865299999997, 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 84 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.345 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.346 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050854+0000", 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1399999999999998e-07 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050855+0000", 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5719999999999999e-06 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050860+0000", 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.347 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050864+0000", 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5750000000000001e-06 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138083+0000", 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087218981000000001 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138226+0000", 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000143545 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.348 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339757+0000", 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20153072 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339856+0000", 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9247999999999996e-05 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.349 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642782+0000", 2024-04-18T21:52:41.350 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30292579400000003 2024-04-18T21:52:41.350 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.350 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.350 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.350 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.350 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.es0 30/26 ECSubRead(tid=10, to_read={3:77cf8780:::smithi09731216-28:head=0,4096,0}, subchunks={3:77cf8780:::smithi09731216-28:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 54.995792590999997, 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30218228800000002, 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:41.351 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.352 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340014+0000", 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.340093+0000", 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8665000000000007e-05 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.440291+0000", 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100198545 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.353 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.641845+0000", 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20155352000000001 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642196+0000", 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035155800000000002 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.354 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.cs7 30/26 ECSubWrite(tid=21, reqid=client.4509.0:101, at_version=30'8, trim_to=0'0, roll_forward_to=30'6))", 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 54.572251412, 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30181939000000002, 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.355 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763555+0000", 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763557+0000", 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.776e-06 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763600+0000", 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3120000000000001e-05 2024-04-18T21:52:41.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.763635+0000", 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5199e-05 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.863748+0000", 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100112399 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.357 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964004+0000", 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10025664500000001 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964030+0000", 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5299999999999998e-05 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065244+0000", 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101214472 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.358 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.065374+0000", 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000130479 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/26 ECSubWriteReply(tid=21, last_complete=35'13, committed=1, applied=1))", 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.957429267000002, 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22690856100000001, 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.359 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378377+0000", 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.360 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378378+0000", 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378380+0000", 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7190000000000001e-06 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378389+0000", 2024-04-18T21:52:41.361 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.378395+0000", 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5210000000000002e-06 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.604185+0000", 2024-04-18T21:52:41.362 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22579012100000001 2024-04-18T21:52:41.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.363 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.363 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.363 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:52.605286+0000", 2024-04-18T21:52:41.363 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011008979999999999 2024-04-18T21:52:41.363 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.363 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.364 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.364 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.364 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:230 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:41.364 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:41.364 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.322537918000002, 2024-04-18T21:52:41.364 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.312981385, 2024-04-18T21:52:41.364 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.365 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.365 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:41.365 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.365 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.365 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.365 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 230 2024-04-18T21:52:41.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.366 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.366 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.366 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.366 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:41.366 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.367 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.367 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013269+0000", 2024-04-18T21:52:41.367 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.367 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013270+0000", 2024-04-18T21:52:41.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5300000000000002e-07 2024-04-18T21:52:41.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.368 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.368 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.368 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013275+0000", 2024-04-18T21:52:41.368 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-04-18T21:52:41.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.369 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013291+0000", 2024-04-18T21:52:41.369 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5466000000000001e-05 2024-04-18T21:52:41.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.369 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.369 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013303+0000", 2024-04-18T21:52:41.370 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2e-05 2024-04-18T21:52:41.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.370 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.370 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.370 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013750+0000", 2024-04-18T21:52:41.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044700300000000002 2024-04-18T21:52:41.371 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.371 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.371 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:41.371 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013767+0000", 2024-04-18T21:52:41.371 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7263999999999999e-05 2024-04-18T21:52:41.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.372 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.372 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225043+0000", 2024-04-18T21:52:41.372 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21127590299999999 2024-04-18T21:52:41.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.225199+0000", 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155716 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.326250+0000", 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10105134 2024-04-18T21:52:41.373 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.374 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.374 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.374 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.374 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:315 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 43.243029841000002, 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70415003200000004, 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 315 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.375 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092778+0000", 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.725e-06 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.376 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092785+0000", 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7950000000000002e-06 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092802+0000", 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6852999999999999e-05 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.377 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092810+0000", 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7950000000000008e-06 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152681+0000", 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059871316000000001 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152755+0000", 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4179999999999998e-05 2024-04-18T21:52:41.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153317+0000", 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00056207900000000005 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253434+0000", 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100116332 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796715+0000", 2024-04-18T21:52:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54328157899999996 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796927+0000", 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021137799999999999 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:316 3.es0 3:7b3c3586:::smithi09731216-12:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 43.242981096999998, 2024-04-18T21:52:41.380 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70474535999999999, 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 316 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092826+0000", 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5999999999999999e-07 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.382 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092827+0000", 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714e-06 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092833+0000", 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0989999999999998e-06 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092836+0000", 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.056e-06 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.383 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153954+0000", 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061118211999999998 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154059+0000", 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010548399999999999 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154712+0000", 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065292300000000002 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.384 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253436+0000", 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098723985 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797355+0000", 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.543919245 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797571+0000", 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021528199999999999 2024-04-18T21:52:41.385 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:317 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 43.242940693999998, 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70519442399999999, 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.386 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 317 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.387 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000002e-07 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092868+0000", 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1440000000000001e-06 2024-04-18T21:52:41.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092873+0000", 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0860000000000001e-06 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192949+0000", 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007585099999999 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252838+0000", 2024-04-18T21:52:41.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059889100000000001 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252927+0000", 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8651000000000002e-05 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253260+0000", 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033289600000000001 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.353758+0000", 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100498198 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797947+0000", 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.444189102 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.798060+0000", 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113109 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.391 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.10s4 41/39 ECSubWrite(tid=6, reqid=client.4509.0:267, at_version=41'14, trim_to=0'0, roll_forward_to=40'10))", 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.630341197999996, 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.305911983, 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705465+0000", 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705468+0000", 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4229999999999999e-06 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705683+0000", 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021499900000000001 2024-04-18T21:52:41.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.705719+0000", 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6758999999999998e-05 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.805843+0000", 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10012386299999999 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806004+0000", 2024-04-18T21:52:41.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160722 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.806040+0000", 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6335999999999999e-05 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011325+0000", 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20528459499999999 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.011377+0000", 2024-04-18T21:52:41.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2286e-05 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:474 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~688128] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.330672373999999, 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31877274300000003, 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.396 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 474 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005134+0000", 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005137+0000", 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000001e-06 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005928+0000", 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00079063400000000002 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005960+0000", 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2727e-05 2024-04-18T21:52:41.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.005976+0000", 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5937e-05 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014929+0000", 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0089524860000000008 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.015108+0000", 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000179366 2024-04-18T21:52:41.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.017762+0000", 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026539900000000002 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106305+0000", 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.088543338999999999 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.323764+0000", 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.217458182 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.323907+0000", 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014321200000000001 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:475 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~540672] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:41.401 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.321728747000002, 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31080646899999997, 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.402 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 475 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014078+0000", 2024-04-18T21:52:41.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014079+0000", 2024-04-18T21:52:41.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.72e-06 2024-04-18T21:52:41.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014732+0000", 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065293199999999995 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014757+0000", 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4352999999999999e-05 2024-04-18T21:52:41.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.014770+0000", 2024-04-18T21:52:41.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.324e-05 2024-04-18T21:52:41.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.020116+0000", 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053463540000000002 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.020404+0000", 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028811800000000002 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.022644+0000", 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022399899999999999 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106309+0000", 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.083664469000000005 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.324710+0000", 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21840135099999999 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.324884+0000", 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017394200000000001 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.409 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:476 3.es0 3:77cf8780:::smithi09731216-28:head [append 0~229376] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.317401536999999, 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30710270099999998, 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.410 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 476 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018405+0000", 2024-04-18T21:52:41.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018408+0000", 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9660000000000001e-06 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018913+0000", 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00050535599999999999 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.018956+0000", 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2740000000000001e-05 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019019+0000", 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2694000000000007e-05 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.023815+0000", 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047959430000000004 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.024087+0000", 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027221100000000001 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.025513+0000", 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014256360000000001 2024-04-18T21:52:41.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106311+0000", 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.080798549999999997 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.325343+0000", 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21903152300000001 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.325508+0000", 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016508199999999999 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:477 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.316748310000001, 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40712708600000003, 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.416 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 477 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019060+0000", 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7859999999999999e-06 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019070+0000", 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0345e-05 2024-04-18T21:52:41.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019098+0000", 2024-04-18T21:52:41.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7392000000000001e-05 2024-04-18T21:52:41.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019115+0000", 2024-04-18T21:52:41.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7765e-05 2024-04-18T21:52:41.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.026938+0000", 2024-04-18T21:52:41.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078220470000000004 2024-04-18T21:52:41.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.027160+0000", 2024-04-18T21:52:41.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000222625 2024-04-18T21:52:41.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.027978+0000", 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081757100000000001 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106314+0000", 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.078336021000000006 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.426038+0000", 2024-04-18T21:52:41.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31972404500000001 2024-04-18T21:52:41.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.426185+0000", 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000147489 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:478 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.316663382000002, 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.709574606, 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.425 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:41.425 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.425 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.425 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.425 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 478 2024-04-18T21:52:41.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.425 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019144+0000", 2024-04-18T21:52:41.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.206e-06 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019151+0000", 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4819999999999999e-06 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019170+0000", 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8906000000000002e-05 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019185+0000", 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5116999999999999e-05 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.028449+0000", 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0092640520000000001 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.028689+0000", 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023998100000000001 2024-04-18T21:52:41.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.427127+0000", 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39843774999999998 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.427384+0000", 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025700499999999998 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.728718+0000", 2024-04-18T21:52:41.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30133410700000002 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:562 3.es0 3:77cf8780:::smithi09731216-28:head [delete] snapc 14=[14,11,10,a,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.018554+0000", 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.317252745999999, 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39250216100000002, 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.430 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 562 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018554+0000", 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018554+0000", 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018557+0000", 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.836e-06 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018568+0000", 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1610000000000001e-05 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018600+0000", 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2060000000000001e-05 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018617+0000", 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7033000000000001e-05 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018756+0000", 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013826299999999999 2024-04-18T21:52:41.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.019238+0000", 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00048212599999999998 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.021858+0000", 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002619864 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.024162+0000", 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023041699999999999 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.410181+0000", 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.38601886000000002 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.411056+0000", 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00087533900000000004 2024-04-18T21:52:41.435 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.436 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.436 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.436 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:578 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:52:41.436 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.743123+0000", 2024-04-18T21:52:41.436 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.592683418, 2024-04-18T21:52:41.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40546945600000001, 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 578 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743123+0000", 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743123+0000", 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743125+0000", 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9259999999999999e-06 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743133+0000", 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6580000000000002e-06 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743151+0000", 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8417e-05 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743161+0000", 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0237999999999999e-05 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743235+0000", 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3946e-05 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743525+0000", 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028927899999999998 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:31.148593+0000", 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40506799199999999 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:579 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:52:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.592621757, 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40650074000000003, 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 579 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743186+0000", 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.16e-07 2024-04-18T21:52:41.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743190+0000", 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5480000000000003e-06 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743199+0000", 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4570000000000005e-06 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743205+0000", 2024-04-18T21:52:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7959999999999998e-06 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.844198+0000", 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100992966 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.844801+0000", 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060208199999999999 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:31.149685+0000", 2024-04-18T21:52:41.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.304884975 2024-04-18T21:52:41.446 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.446 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.446 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.446 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:580 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:52:41.446 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.592590827, 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40696587899999997, 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 580 2024-04-18T21:52:41.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4799999999999998e-07 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743220+0000", 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5829999999999998e-06 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743227+0000", 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.379e-06 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743233+0000", 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9220000000000002e-06 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.937914+0000", 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19468091200000001 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.938494+0000", 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000580282 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:31.150182+0000", 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21168725299999999 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.es0 82/78 ECSubReadReply(tid=84, attrs_read=0))", 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:39.415574+0000", 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 1.920232589, 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.277578198, 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:41.451 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:39.415574+0000", 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:39.415574+0000", 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:39.415575+0000", 2024-04-18T21:52:41.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4000000000000001e-07 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:39.415698+0000", 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012296599999999999 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:39.415724+0000", 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6948999999999999e-05 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:41.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:39.415738+0000", 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3903999999999999e-05 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:39.693119+0000", 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.27738083499999999 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:39.693152+0000", 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2904000000000002e-05 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.454 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.455 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.455 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:41.455 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:41.455 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:41.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:41.456 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:41.456+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:41.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:41.557 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:41.554+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:41.636 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:41.636 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:41.637 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:41.637 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:41.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:41.652 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:41.652 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:41.658 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:41.655+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:41.669 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:41.669 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:41.686 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:41.686 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:41.705 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:41.705 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:41.760 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:41.755+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:41.768 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:41.768 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:41.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:41.823 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:41.823 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:41.848 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:41.848 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:41.860 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:41.855+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:41.873 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:41.874 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:41.900 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:41.900 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:41.923 INFO:tasks.rados.rados.0.smithi097.stdout:252: finishing write tid 1 to smithi09731216-20 2024-04-18T21:52:41.927 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:41.927 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:41.942 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:41.942 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:41.950 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:41.951 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:41.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:41.960 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:41.954+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:41.975 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:41.975 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:41.999 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:41.999 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:42.024 INFO:tasks.rados.rados.0.smithi097.stdout:252: finishing write tid 2 to smithi09731216-20 2024-04-18T21:52:42.061 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:42.059+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:42.091 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:42.091 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:42.105 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:42.162 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:42.159+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:42.229 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:42.229 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:42.244 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:42.293 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:42.290+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:42.325 INFO:tasks.rados.rados.0.smithi097.stdout:252: finishing write tid 3 to smithi09731216-20 2024-04-18T21:52:42.365 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:42.363+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:42.371 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:42.371 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:42.385 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:42.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-18T21:52:42.466 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:42.463+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:42.555 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:42.555 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:42.567 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:42.563+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:42.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:42.615 INFO:tasks.rados.rados.0.smithi097.stdout:256: finishing write tid 1 to smithi09731216-50 2024-04-18T21:52:42.631 INFO:tasks.rados.rados.0.smithi097.stdout:252: finishing write tid 4 to smithi09731216-20 2024-04-18T21:52:42.668 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:42.662+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:42.693 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:42.693 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:42.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:42.793 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:42.790+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:42.820 INFO:tasks.rados.rados.0.smithi097.stdout:256: finishing write tid 2 to smithi09731216-50 2024-04-18T21:52:42.820 INFO:tasks.rados.rados.0.smithi097.stdout:256: finishing write tid 3 to smithi09731216-50 2024-04-18T21:52:42.820 INFO:tasks.rados.rados.0.smithi097.stdout:256: finishing write tid 4 to smithi09731216-50 2024-04-18T21:52:42.820 INFO:tasks.rados.rados.0.smithi097.stdout:256: finishing write tid 5 to smithi09731216-50 2024-04-18T21:52:42.836 INFO:tasks.rados.rados.0.smithi097.stdout:252: finishing write tid 5 to smithi09731216-20 2024-04-18T21:52:42.836 INFO:tasks.rados.rados.0.smithi097.stdout:252: oid 20 updating version 0 to 134 2024-04-18T21:52:42.836 INFO:tasks.rados.rados.0.smithi097.stdout:252: oid 20 version 134 is already newer than 133 2024-04-18T21:52:42.836 INFO:tasks.rados.rados.0.smithi097.stdout:252: oid 20 updating version 134 to 135 2024-04-18T21:52:42.836 INFO:tasks.rados.rados.0.smithi097.stdout:252: oid 20 version 135 is already newer than 132 2024-04-18T21:52:42.836 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 20 v 135 (ObjNum 107 snap 19 seq_num 107) dirty exists 2024-04-18T21:52:42.837 INFO:tasks.rados.rados.0.smithi097.stdout:252: left oid 20 (ObjNum 107 snap 19 seq_num 107) 2024-04-18T21:52:42.869 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:42.867+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:42.874 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:42.874 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:42.888 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:42.970 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:42.967+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:43.054 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:43.054 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:43.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:43.071 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:43.066+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:43.171 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:43.167+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:43.196 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:43.196 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:43.210 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:43.230 INFO:tasks.rados.rados.0.smithi097.stdout:256: finishing write tid 6 to smithi09731216-50 2024-04-18T21:52:43.230 INFO:tasks.rados.rados.0.smithi097.stdout:256: oid 50 updating version 0 to 23 2024-04-18T21:52:43.230 INFO:tasks.rados.rados.0.smithi097.stdout:256: oid 50 updating version 23 to 24 2024-04-18T21:52:43.230 INFO:tasks.rados.rados.0.smithi097.stdout:256: oid 50 updating version 24 to 27 2024-04-18T21:52:43.231 INFO:tasks.rados.rados.0.smithi097.stdout:256: oid 50 version 27 is already newer than 25 2024-04-18T21:52:43.231 INFO:tasks.rados.rados.0.smithi097.stdout:256: oid 50 version 27 is already newer than 26 2024-04-18T21:52:43.231 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 50 v 27 (ObjNum 108 snap 21 seq_num 108) dirty exists 2024-04-18T21:52:43.231 INFO:tasks.rados.rados.0.smithi097.stdout:256: left oid 50 (ObjNum 108 snap 21 seq_num 108) 2024-04-18T21:52:43.233 INFO:tasks.rados.rados.0.smithi097.stdout:252: done (3 left) 2024-04-18T21:52:43.233 INFO:tasks.rados.rados.0.smithi097.stdout:256: done (2 left) 2024-04-18T21:52:43.233 INFO:tasks.rados.rados.0.smithi097.stdout:257: done (1 left) 2024-04-18T21:52:43.233 INFO:tasks.rados.rados.0.smithi097.stdout:259: done (0 left) 2024-04-18T21:52:43.233 INFO:tasks.rados.rados.0.smithi097.stdout:260: rmattr oid 38 current snap is 21 2024-04-18T21:52:43.233 INFO:tasks.rados.rados.0.smithi097.stdout:260: done (0 left) 2024-04-18T21:52:43.233 INFO:tasks.rados.rados.0.smithi097.stdout:261: snap_create 2024-04-18T21:52:43.272 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:43.270+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:43.324 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:43.318+0000 7f475a357640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 6 after trimming was started 2024-04-18T21:52:43.339 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:43.339 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:43.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:43.373 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:43.371+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:43.408 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is set 2024-04-18T21:52:43.413 INFO:tasks.rados.rados.0.smithi097.stdout:261: done (0 left) 2024-04-18T21:52:43.413 INFO:tasks.rados.rados.0.smithi097.stdout:262: copy_from oid 48 from oid 47 current snap is 22 2024-04-18T21:52:43.413 INFO:tasks.rados.rados.0.smithi097.stdout:263: read oid 34 snap -1 2024-04-18T21:52:43.413 INFO:tasks.rados.rados.0.smithi097.stdout:263: expect (ObjNum 94 snap 18 seq_num 94) 2024-04-18T21:52:43.413 INFO:tasks.rados.rados.0.smithi097.stdout:264: snap_create 2024-04-18T21:52:43.481 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:43.479+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:43.528 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:43.528 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:43.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:43.575 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:43.571+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:43.669 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:43.669 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:43.676 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:43.671+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:43.683 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:43.777 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:43.774+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:43.802 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:43.802 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:43.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:43.878 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:43.874+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:43.948 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:43.948 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:43.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:44.009 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:44.007+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:44.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:44.079 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:44.074+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:44.086 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:44.086 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:44.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:44.137 INFO:tasks.rados.rados.0.smithi097.stdout:263: expect (ObjNum 94 snap 18 seq_num 94) 2024-04-18T21:52:44.176 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:44.176 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:44.179 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:44.175+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:44.193 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:44.193 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:44.209 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:44.209 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:44.226 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:44.226 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:44.248 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:44.248 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:44.260 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:44.260 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:44.269 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:44.269 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:44.275 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:44.281 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:44.278+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:44.290 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:44.290 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:44.311 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:44.312 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:44.331 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:44.331 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:44.351 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:44.351 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:44.372 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:44.372 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:44.382 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:44.378+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:44.394 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:44.394 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:44.399 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:44.399 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:44.399 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:44.399 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:44.401 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:44.401 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:52:44.402 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:44.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-04-18T21:52:44.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:44.419 INFO:tasks.rados.rados.0.smithi097.stdout:263: done (2 left) 2024-04-18T21:52:44.419 INFO:tasks.rados.rados.0.smithi097.stdout:264: done (1 left) 2024-04-18T21:52:44.419 INFO:tasks.rados.rados.0.smithi097.stdout:265: read oid 41 snap -1 2024-04-18T21:52:44.419 INFO:tasks.rados.rados.0.smithi097.stdout:265: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-18T21:52:44.419 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 12 current snap is 23 2024-04-18T21:52:44.419 INFO:tasks.rados.rados.0.smithi097.stdout:266: seq_num 110 ranges {2932736=688128,3620864=507904,4128768=655360} 2024-04-18T21:52:44.445 INFO:tasks.rados.rados.0.smithi097.stdout:266: writing smithi09731216-12 from 2932736 to 3620864 tid 1 2024-04-18T21:52:44.448 INFO:tasks.rados.rados.0.smithi097.stdout:266: writing smithi09731216-12 from 3620864 to 4128768 tid 2 2024-04-18T21:52:44.453 INFO:tasks.rados.rados.0.smithi097.stdout:266: writing smithi09731216-12 from 4128768 to 4784128 tid 3 2024-04-18T21:52:44.453 INFO:tasks.rados.rados.0.smithi097.stdout:267: read oid 43 snap 8 2024-04-18T21:52:44.453 INFO:tasks.rados.rados.0.smithi097.stdout:267: expect (ObjNum 74 snap 8 seq_num 74) 2024-04-18T21:52:44.453 INFO:tasks.rados.rados.0.smithi097.stdout:268: read oid 18 snap -1 2024-04-18T21:52:44.453 INFO:tasks.rados.rados.0.smithi097.stdout:268: expect deleted 2024-04-18T21:52:44.453 INFO:tasks.rados.rados.0.smithi097.stdout:269: read oid 22 snap 21 2024-04-18T21:52:44.453 INFO:tasks.rados.rados.0.smithi097.stdout:269: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-18T21:52:44.453 INFO:tasks.rados.rados.0.smithi097.stdout:270: snap_remove snap 20 2024-04-18T21:52:44.482 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:44.479+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:44.541 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:44.541 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:44.541 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:44.541 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:44.541 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:44.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:44.583 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:44.578+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:44.684 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:44.678+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:44.685 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:44.685 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:44.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=7, reqid=client.4509.0:94, at_version=30'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.022587414999997, 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201169337, 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661077+0000", 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4600000000000002e-06 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661394+0000", 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031720499999999998 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661424+0000", 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9822999999999999e-05 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661441+0000", 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7496e-05 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661661+0000", 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000219602 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661676+0000", 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5333e-05 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862187+0000", 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200510299 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862244+0000", 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7119e-05 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.022184107999998, 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201435581, 2024-04-18T21:52:44.690 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661479+0000", 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0839999999999999e-06 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661509+0000", 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0389999999999999e-05 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661529+0000", 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0219000000000001e-05 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661553+0000", 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3836999999999999e-05 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861851+0000", 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20029786299999999 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861866+0000", 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4769999999999999e-05 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862836+0000", 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097006600000000001 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862913+0000", 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7352000000000001e-05 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.8s4 30/26 ECSubWrite(tid=12, reqid=client.4509.0:104, at_version=30'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:44.694 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 57.814103807000002, 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11605781900000001, 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869558+0000", 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869560+0000", 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-06 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869593+0000", 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3216000000000002e-05 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.869624+0000", 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1164000000000002e-05 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969744+0000", 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100119518 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969958+0000", 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021381800000000001 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.969985+0000", 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7376999999999999e-05 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985510+0000", 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015525357 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985616+0000", 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105479 2024-04-18T21:52:44.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.17s4 32/31 ECSubWrite(tid=1, reqid=client.4509.0:119, at_version=32'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.188533247999999, 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18978218799999999, 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495129+0000", 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495130+0000", 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0860000000000001e-06 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495154+0000", 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4457999999999999e-05 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495174+0000", 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9426999999999999e-05 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495188+0000", 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4813e-05 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683568+0000", 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.188379871 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.683583+0000", 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5227e-05 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.684845+0000", 2024-04-18T21:52:44.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012619560000000001 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.684911+0000", 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5350000000000003e-05 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.14s6 32/31 ECSubWrite(tid=20, reqid=client.4509.0:125, at_version=32'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.087157097999999, 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20154013100000001, 2024-04-18T21:52:44.703 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3799999999999997e-07 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596521+0000", 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5455999999999999e-05 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596532+0000", 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0881e-05 2024-04-18T21:52:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696606+0000", 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100073932 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796760+0000", 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10015476600000001 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796771+0000", 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0278999999999999e-05 2024-04-18T21:52:44.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797921+0000", 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001150657 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.798045+0000", 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012362199999999999 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.707 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es6 41/26 ECSubWrite(tid=30, reqid=client.4509.0:316, at_version=41'18, trim_to=0'0, roll_forward_to=38'15))", 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 46.487761243000001, 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200900518, 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195902+0000", 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3699999999999999e-07 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195937+0000", 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5682000000000002e-05 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195960+0000", 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2940999999999999e-05 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296055+0000", 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100094687 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296167+0000", 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011265900000000001 2024-04-18T21:52:44.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296183+0000", 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5605000000000001e-05 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.396721+0000", 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10053751700000001 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.396801+0000", 2024-04-18T21:52:44.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0489999999999997e-05 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.19s5 41/31 ECSubWrite(tid=38, reqid=client.4509.0:318, at_version=41'9, trim_to=0'0, roll_forward_to=33'7))", 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 46.308372005999999, 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.199621414, 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375290+0000", 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375292+0000", 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7069999999999999e-06 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375331+0000", 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9774999999999998e-05 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375361+0000", 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.974e-05 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375379+0000", 2024-04-18T21:52:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8017e-05 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375529+0000", 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015042500000000001 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.375547+0000", 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7414999999999998e-05 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.574854+0000", 2024-04-18T21:52:44.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19930687499999999 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.574911+0000", 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7460000000000002e-05 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.17s4,RELEASE,e:42,from:8(0),reservation_nonce:0)", 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 44.896397774, 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20038272400000001, 2024-04-18T21:52:44.716 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787265+0000", 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.02e-06 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787274+0000", 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.439e-06 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787295+0000", 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1121e-05 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787312+0000", 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7838000000000001e-05 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987569+0000", 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200256506 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987603+0000", 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4072000000000002e-05 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987647+0000", 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3727999999999999e-05 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.11s3 47/45 ECSubRead(tid=45, to_read={3:8fad7925:::smithi09731216-43:head=0,262144,0}, subchunks={3:8fad7925:::smithi09731216-43:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 41.285925145999997, 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20153923400000001, 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.720 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397739+0000", 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.367e-06 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397761+0000", 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1525000000000001e-05 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397804+0000", 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3488000000000001e-05 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.497972+0000", 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100167611 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.598288+0000", 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100316505 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.599276+0000", 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00098773799999999994 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.723 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 35.696944780999999, 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20394542500000001, 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986718+0000", 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087949+0000", 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10123096600000001 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087978+0000", 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9116999999999999e-05 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188141+0000", 2024-04-18T21:52:44.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10016269799999999 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.190504+0000", 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023637250000000001 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.190662+0000", 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000158076 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 53/51 ECSubWriteReply(tid=1, last_complete=53'7, committed=1, applied=1))", 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 35.165447194000002, 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.180820172, 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518215+0000", 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518217+0000", 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9199999999999998e-06 2024-04-18T21:52:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518227+0000", 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0356e-05 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518255+0000", 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8144e-05 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.518274+0000", 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9218000000000001e-05 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.697313+0000", 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17903920600000001 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.699035+0000", 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017213280000000001 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 64/58 ECSubWrite(tid=206, reqid=osd.7.0:206, at_version=64'51, trim_to=0'0, roll_forward_to=62'47))", 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.492846+0000", 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.190816225999999, 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20166795100000001, 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492846+0000", 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492846+0000", 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492846+0000", 2024-04-18T21:52:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9100000000000003e-07 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492878+0000", 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1483999999999999e-05 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492892+0000", 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4476e-05 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.592975+0000", 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100083016 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.693140+0000", 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100164527 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.693156+0000", 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6273000000000002e-05 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.694448+0000", 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001292039 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.694514+0000", 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5345000000000005e-05 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 64/58 ECSubWrite(tid=208, reqid=osd.7.0:206, at_version=0'0, trim_to=0'0, roll_forward_to=64'51))", 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.885368+0000", 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 23.798294213999998, 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20183865500000001, 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885368+0000", 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885368+0000", 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885369+0000", 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3340000000000001e-06 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885408+0000", 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9511000000000002e-05 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885436+0000", 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7804999999999999e-05 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.985514+0000", 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100078234 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.985605+0000", 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0080999999999994e-05 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.985617+0000", 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.27e-05 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.087123+0000", 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101505472 2024-04-18T21:52:44.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.087206+0000", 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3517999999999996e-05 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 68/58 ECSubWrite(tid=215, reqid=osd.7.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.920355+0000", 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.763307231999999, 2024-04-18T21:52:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20283171999999999, 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920355+0000", 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920355+0000", 2024-04-18T21:52:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920357+0000", 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3659999999999999e-06 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920382+0000", 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5085000000000001e-05 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920413+0000", 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1105000000000001e-05 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920430+0000", 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6668999999999999e-05 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.022477+0000", 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102047579 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.022493+0000", 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5625999999999999e-05 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.123112+0000", 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100619477 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.123186+0000", 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3813000000000006e-05 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.745 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.13s7,QREQUEST,e:68,from:7(0),reservation_nonce:7)", 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:25.324753+0000", 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.358909154999999, 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20066576, 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324753+0000", 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324753+0000", 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324755+0000", 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0269999999999998e-06 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324764+0000", 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0219999999999993e-06 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324794+0000", 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9802e-05 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.424947+0000", 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10015381900000001 2024-04-18T21:52:44.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.525281+0000", 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10033357 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.525321+0000", 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0108000000000003e-05 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.525418+0000", 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7411999999999995e-05 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 73/58 ECSubWrite(tid=218, reqid=client.4509.0:563, at_version=73'59, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.258690026, 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30546464400000001, 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424974+0000", 2024-04-18T21:52:44.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9e-06 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425568+0000", 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00059416700000000005 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425597+0000", 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9305000000000001e-05 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425613+0000", 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5449000000000001e-05 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.625991+0000", 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200377891 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.626011+0000", 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0956000000000001e-05 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730378+0000", 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104366235 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730436+0000", 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8740999999999999e-05 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 73/58 ECSubWrite(tid=219, reqid=client.4509.0:564, at_version=73'60, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.248350901, 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29521209100000001, 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435313+0000", 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.807e-06 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435737+0000", 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042456200000000002 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435778+0000", 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0555e-05 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.536874+0000", 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101095886 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.727552+0000", 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19067846699999999 2024-04-18T21:52:44.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.727571+0000", 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8941999999999999e-05 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730493+0000", 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029220800000000001 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730523+0000", 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9791999999999999e-05 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 74/58 ECSubWrite(tid=221, reqid=client.4509.0:565, at_version=74'61, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.540785+0000", 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.142877164, 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18979601099999999, 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.540785+0000", 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.540785+0000", 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.540785+0000", 2024-04-18T21:52:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5299999999999999e-07 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.541333+0000", 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00054812299999999997 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.541380+0000", 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6814999999999998e-05 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.541421+0000", 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0988000000000001e-05 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.728666+0000", 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18724543599999999 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.728686+0000", 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9182999999999999e-05 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730555+0000", 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001868895 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730581+0000", 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6118e-05 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.763 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.as2 74/26 ECSubWrite(tid=61, reqid=client.4509.0:572, at_version=74'17, trim_to=0'0, roll_forward_to=70'15))", 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.639729+0000", 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.043932578, 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200655103, 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.639729+0000", 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.639729+0000", 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.639731+0000", 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.866e-06 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.640108+0000", 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00037644799999999998 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.640135+0000", 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7212e-05 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.640152+0000", 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6943e-05 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.736587+0000", 2024-04-18T21:52:44.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.096435114000000002 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.736603+0000", 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5719e-05 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.840260+0000", 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10365738400000001 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.840384+0000", 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012441700000000001 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.as2 74/26 ECSubWrite(tid=62, reqid=client.4509.0:573, at_version=74'18, trim_to=0'0, roll_forward_to=70'15))", 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.645106+0000", 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.038555840000001, 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.195330792, 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:44.768 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.645106+0000", 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.645106+0000", 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.645107+0000", 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5200000000000001e-06 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.645399+0000", 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029101300000000001 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.645428+0000", 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9068e-05 2024-04-18T21:52:44.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.645443+0000", 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5610000000000001e-05 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.837997+0000", 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19255393800000001 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:44.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.838013+0000", 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6291999999999999e-05 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.840400+0000", 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023864759999999998 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.840437+0000", 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6875000000000002e-05 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.772 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:44.773 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:44.773 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:44.773 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-18T21:52:44.707+0000 7f0412e7b640 -1 mon.a@0(leader).osd e88 definitely_dead 0 2024-04-18T21:52:44.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:44.785 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:44.782+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:44.886 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:44.883+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:44.900 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:44.900 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:44.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:45.001 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:44.999+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:45.079 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:45.079 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:45.079 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:45.080 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:45.080 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:45.087 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:45.083+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:45.093 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:45.188 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:45.182+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:45.256 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:97 3.4s0 3:209f2930:::smithi09731216-33:head [append 0~540672,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.268151713000002, 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4026191889999999, 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985858+0000", 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.92e-06 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987540+0000", 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001681463 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987566+0000", 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6043000000000001e-05 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987586+0000", 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9930000000000001e-05 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087950+0000", 2024-04-18T21:52:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100363824 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087956+0000", 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9200000000000001e-06 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388374+0000", 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3004186340000001 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388475+0000", 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100455 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:98 3.4s0 3:209f2930:::smithi09731216-33:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.266390311000002, 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.401002842, 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 98 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987618+0000", 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5e-06 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987625+0000", 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6769999999999998e-06 2024-04-18T21:52:45.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987637+0000", 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1693e-05 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987646+0000", 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3700000000000001e-06 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088140+0000", 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10049390900000001 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088145+0000", 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3600000000000004e-06 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388530+0000", 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300384269 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388620+0000", 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0063999999999997e-05 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:99 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.266347570999997, 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503241456, 2024-04-18T21:52:45.267 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987661+0000", 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5999999999999991e-07 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987664+0000", 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6459999999999999e-06 2024-04-18T21:52:45.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987675+0000", 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0967e-05 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987684+0000", 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6929999999999996e-06 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088241+0000", 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100557049 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088244+0000", 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2729999999999998e-06 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388746+0000", 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300502061 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388878+0000", 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131533 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490901+0000", 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102023274 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.272 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:117 3.bs0 3:d4cd3926:::smithi09731216-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.066278562000001, 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.520974363, 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 117 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187730+0000", 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.127e-06 2024-04-18T21:52:45.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187735+0000", 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.536e-06 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187752+0000", 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6186999999999999e-05 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187762+0000", 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0587e-05 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281120+0000", 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093357525999999996 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281296+0000", 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017659300000000001 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405205+0000", 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.123908465 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405449+0000", 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000244505 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708703+0000", 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30325383700000003 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:126 3.14s0 3:2c76b789:::smithi09731216-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.74775005, 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50118609800000002, 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.278 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 126 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506258+0000", 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506267+0000", 2024-04-18T21:52:45.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7159999999999992e-06 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506287+0000", 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9542999999999999e-05 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506301+0000", 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4572999999999999e-05 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511483+0000", 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051813989999999997 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511616+0000", 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013376499999999999 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803750+0000", 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29213323699999999 2024-04-18T21:52:45.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803998+0000", 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024861799999999998 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.007443+0000", 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20344494699999999 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.16s4 32/31 ECSubWrite(tid=15, reqid=client.4509.0:110, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.165850935000002, 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.42111731499999999, 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088157+0000", 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088207+0000", 2024-04-18T21:52:45.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0584999999999998e-05 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088241+0000", 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3784e-05 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088269+0000", 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8314999999999999e-05 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408130+0000", 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31986045299999999 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408140+0000", 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0248000000000001e-05 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509225+0000", 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10108521500000001 2024-04-18T21:52:45.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509274+0000", 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8374999999999997e-05 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:216 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.238094555000004, 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51394664400000001, 2024-04-18T21:52:45.288 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015916+0000", 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0130000000000002e-06 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015930+0000", 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4705000000000001e-05 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015964+0000", 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4072000000000002e-05 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015996+0000", 2024-04-18T21:52:45.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1198999999999997e-05 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016142+0000", 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014630900000000001 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016155+0000", 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2581e-05 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.020038+0000", 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038838789999999998 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.020191+0000", 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000152559 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.021481+0000", 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012896680000000001 2024-04-18T21:52:45.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128155+0000", 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106673879 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.529681+0000", 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40152656199999998 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.529859+0000", 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017821800000000001 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:217 3.bs0 3:d90895f4:::smithi09731216-9:head [rollback 2] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.237957520999998, 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51416292299999999, 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.295 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016051+0000", 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6470000000000001e-06 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016062+0000", 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0451999999999999e-05 2024-04-18T21:52:45.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016085+0000", 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2776e-05 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016103+0000", 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8131e-05 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016278+0000", 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017520200000000001 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016289+0000", 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1048e-05 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.023057+0000", 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067683750000000001 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.023334+0000", 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027651500000000002 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.024976+0000", 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016423379999999999 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128157+0000", 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10318049899999999 2024-04-18T21:52:45.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.530073+0000", 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40191629299999998 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.530213+0000", 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013964700000000001 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:218 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.237874216000002, 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61382267800000001, 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 218 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.302 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016134+0000", 2024-04-18T21:52:45.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016141+0000", 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5719999999999999e-06 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016161+0000", 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9814e-05 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016177+0000", 2024-04-18T21:52:45.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6028e-05 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016365+0000", 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018808400000000001 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016384+0000", 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-05 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026326+0000", 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0099418909999999996 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026591+0000", 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026506899999999999 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.027603+0000", 2024-04-18T21:52:45.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010121799999999999 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128223+0000", 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100619874 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629817+0000", 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50159395299999998 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629956+0000", 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013901300000000001 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:298 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~720896] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 47.192536672999999, 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.422019223, 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.308 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 298 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061474+0000", 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2770000000000001e-06 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063120+0000", 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001645903 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063156+0000", 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.642e-05 2024-04-18T21:52:45.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063174+0000", 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7621e-05 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063316+0000", 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000141753 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063330+0000", 2024-04-18T21:52:45.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4324999999999999e-05 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.166460+0000", 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103130208 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.166636+0000", 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175792 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.170069+0000", 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003432778 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279315+0000", 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.109246564 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.483316+0000", 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20400066 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.483490+0000", 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017392199999999999 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:299 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 47.189692491000002, 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.518657543, 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.314 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 299 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064317+0000", 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5900000000000002e-06 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065148+0000", 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000830709 2024-04-18T21:52:45.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065184+0000", 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5738999999999998e-05 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065209+0000", 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5627e-05 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065508+0000", 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029824799999999998 2024-04-18T21:52:45.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065539+0000", 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0871999999999999e-05 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.172826+0000", 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.107287676 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.174217+0000", 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001390891 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.177064+0000", 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028466120000000001 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279319+0000", 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102254998 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.582844+0000", 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30352516899999998 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.582972+0000", 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000128412 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.320 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:300 3.bs0 3:d90895f4:::smithi09731216-9:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 47.188747898000003, 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.71857623199999998, 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 300 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065261+0000", 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9690000000000001e-06 2024-04-18T21:52:45.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065272+0000", 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0545e-05 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065299+0000", 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6927000000000002e-05 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065319+0000", 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0035e-05 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165723+0000", 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100404099 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165747+0000", 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.438e-05 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279995+0000", 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.114247892 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280276+0000", 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028071299999999998 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281155+0000", 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087942000000000001 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.381976+0000", 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100820639 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783683+0000", 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40170656199999999 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783836+0000", 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000153051 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:301 3.bs0 3:d90895f4:::smithi09731216-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 47.188615744000003, 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.82307189599999997, 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.327 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 301 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.328 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065393+0000", 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.553e-06 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065401+0000", 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9719999999999992e-06 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065425+0000", 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3547e-05 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065447+0000", 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2234000000000001e-05 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165843+0000", 2024-04-18T21:52:45.330 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100395994 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165860+0000", 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6871000000000002e-05 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281656+0000", 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11579645099999999 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281793+0000", 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013725 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783958+0000", 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50216431100000003 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.784112+0000", 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015411799999999999 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.332 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.888463+0000", 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10435159500000001 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:422 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~294912] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.807127292000001, 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61132010699999995, 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.333 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 422 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.334 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446882+0000", 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5459999999999998e-06 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447318+0000", 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00043564599999999998 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447349+0000", 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0888e-05 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447365+0000", 2024-04-18T21:52:45.336 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5593000000000001e-05 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447499+0000", 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013438999999999999 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447509+0000", 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6509999999999996e-06 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.337 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.556315+0000", 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10880680299999999 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.556603+0000", 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028725400000000002 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.338 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.558288+0000", 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001684786 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638308+0000", 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.080020547999999997 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.339 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058105+0000", 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41979719399999998 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058200+0000", 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4808000000000004e-05 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.340 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:423 3.2s0 3:44bf1f18:::smithi09731216-13:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.806610149000001, 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61117605200000003, 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 423 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.341 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447398+0000", 2024-04-18T21:52:45.342 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.353e-06 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447416+0000", 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7353000000000001e-05 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447431+0000", 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5645e-05 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.343 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447441+0000", 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2690000000000005e-06 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447569+0000", 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012837999999999999 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447573+0000", 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3270000000000002e-06 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.559335+0000", 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11176207000000001 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.559597+0000", 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026142 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.345 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.560692+0000", 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010952799999999999 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638310+0000", 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.077617829999999999 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058483+0000", 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.42017316199999999 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.346 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058573+0000", 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9963000000000006e-05 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:424 3.2s0 3:44bf1f18:::smithi09731216-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.806512055999999, 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.81405277300000001, 2024-04-18T21:52:45.347 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 424 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447496+0000", 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1699999999999997e-07 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.349 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447501+0000", 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5059999999999997e-06 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447513+0000", 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2592000000000001e-05 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447526+0000", 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2309e-05 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.350 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.547672+0000", 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10014685800000001 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.547684+0000", 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1232e-05 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.756870+0000", 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.209186237 2024-04-18T21:52:45.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.756965+0000", 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5280999999999998e-05 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.059024+0000", 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30205871899999998 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.059128+0000", 2024-04-18T21:52:45.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010387600000000001 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.261548+0000", 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.202420246 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:434 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:45.353 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.80646368, 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.58875951599999998, 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 434 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:45.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999998e-07 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447549+0000", 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7740000000000004e-06 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447560+0000", 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0522e-05 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447569+0000", 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.312e-06 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447614+0000", 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5370000000000001e-05 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447620+0000", 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.502e-06 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638433+0000", 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.190813609 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.357 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638582+0000", 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014889800000000001 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.036303+0000", 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39772072600000002 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:435 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.806417257, 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.58911970899999999, 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.359 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 435 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.361 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447591+0000", 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2270000000000001e-06 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447598+0000", 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6630000000000004e-06 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447616+0000", 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8587999999999999e-05 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447631+0000", 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4326e-05 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447673+0000", 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2159000000000001e-05 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447677+0000", 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0799999999999999e-06 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639051+0000", 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19137447299999999 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639173+0000", 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121443 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.036710+0000", 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39753674999999999 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:436 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.806355015000001, 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.59017562599999995, 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.366 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 436 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.367 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447653+0000", 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1540000000000001e-06 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447661+0000", 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7449999999999995e-06 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.368 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447679+0000", 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8263000000000001e-05 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447693+0000", 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3642000000000001e-05 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447729+0000", 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6124000000000002e-05 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.369 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447733+0000", 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9620000000000004e-06 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639451+0000", 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19171812099999999 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.370 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639522+0000", 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0956000000000003e-05 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.037828+0000", 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39830565899999998 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(osd.1.19:2 3.bs0 3:d688e784:::smithi09731216-24:head [assert-version v39,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:45.371 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.518729+0000", 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.735277882999998, 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41734706799999999, 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518729+0000", 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.372 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518729+0000", 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518730+0000", 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.094e-06 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518735+0000", 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.617e-06 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.373 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518744+0000", 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2709999999999998e-06 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518753+0000", 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7090000000000008e-06 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518874+0000", 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012108399999999999 2024-04-18T21:52:45.374 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.518889+0000", 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4636e-05 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.538447+0000", 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019558351000000002 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.375 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.538594+0000", 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014643600000000001 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.936076+0000", 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39748286999999999 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.376 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:45.378 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:45.379 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:45.375+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:45.418 INFO:teuthology.orchestra.run.smithi097.stderr:marked out osd.7. 2024-04-18T21:52:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:270: done (6 left) 2024-04-18T21:52:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:271: rollback oid 33 current snap is 23 2024-04-18T21:52:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 33 to 21 2024-04-18T21:52:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:272: read oid 29 snap -1 2024-04-18T21:52:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:272: expect deleted 2024-04-18T21:52:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 49 current snap is 23 2024-04-18T21:52:45.434 INFO:tasks.rados.rados.0.smithi097.stdout:273: seq_num 111 ranges {0=704512,704512=655360,1359872=507904} 2024-04-18T21:52:45.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:52:45.439 INFO:tasks.rados.rados.0.smithi097.stdout:273: writing smithi09731216-49 from 0 to 704512 tid 1 2024-04-18T21:52:45.443 INFO:tasks.rados.rados.0.smithi097.stdout:273: writing smithi09731216-49 from 704512 to 1359872 tid 2 2024-04-18T21:52:45.446 INFO:tasks.rados.rados.0.smithi097.stdout:273: writing smithi09731216-49 from 1359872 to 1867776 tid 3 2024-04-18T21:52:45.446 INFO:tasks.rados.rados.0.smithi097.stdout:274: setattr oid 30 current snap is 23 2024-04-18T21:52:45.446 INFO:tasks.rados.rados.0.smithi097.stdout:267: expect (ObjNum 74 snap 8 seq_num 74) 2024-04-18T21:52:45.478 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:45.475+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:45.514 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:45.514 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:45.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s5 88/87 ECSubWrite(tid=272, reqid=osd.7.0:272, at_version=88'73, trim_to=0'0, roll_forward_to=88'71))", 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:45.342943+0000", 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.17227097499999999, 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17231447, 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.342943+0000", 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.342943+0000", 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.342944+0000", 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6169999999999999e-06 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.342983+0000", 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8065000000000002e-05 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.343012+0000", 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9765999999999999e-05 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.343028+0000", 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5308999999999999e-05 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s5 88/87 ECSubWrite(tid=271, reqid=osd.7.0:270, at_version=0'0, trim_to=0'0, roll_forward_to=88'71))", 2024-04-18T21:52:45.517 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:45.340147+0000", 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.17506698200000001, 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17519964800000001, 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.340147+0000", 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.340147+0000", 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.340149+0000", 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0360000000000001e-06 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.340173+0000", 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3743e-05 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.340214+0000", 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1734999999999999e-05 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.340237+0000", 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2478999999999999e-05 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.440456+0000", 2024-04-18T21:52:45.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100218999 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:45.440470+0000", 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4388e-05 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 2 2024-04-18T21:52:45.521 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:45.523 INFO:tasks.rados.rados.0.smithi097.stdout:262: finishing copy_from to smithi09731216-48 2024-04-18T21:52:45.523 INFO:tasks.rados.rados.0.smithi097.stdout:262: got expected ENOENT (src dne) 2024-04-18T21:52:45.523 INFO:tasks.rados.rados.0.smithi097.stdout:266: finishing write tid 1 to smithi09731216-12 2024-04-18T21:52:45.523 INFO:tasks.rados.rados.0.smithi097.stdout:262: finishing copy_from racing read to smithi09731216-48 2024-04-18T21:52:45.523 INFO:tasks.rados.rados.0.smithi097.stdout:266: finishing write tid 2 to smithi09731216-12 2024-04-18T21:52:45.523 INFO:tasks.rados.rados.0.smithi097.stdout:269: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-18T21:52:45.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:45.533 INFO:tasks.rados.rados.0.smithi097.stdout:262: done (9 left) 2024-04-18T21:52:45.534 INFO:tasks.rados.rados.0.smithi097.stdout:267: done (8 left) 2024-04-18T21:52:45.534 INFO:tasks.rados.rados.0.smithi097.stdout:268: done (7 left) 2024-04-18T21:52:45.534 INFO:tasks.rados.rados.0.smithi097.stdout:269: done (6 left) 2024-04-18T21:52:45.534 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 22 current snap is 23 2024-04-18T21:52:45.534 INFO:tasks.rados.rados.0.smithi097.stdout:275: seq_num 112 ranges {360448=720896,1081344=802816,1884160=360448} 2024-04-18T21:52:45.536 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:45.534+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:45.546 INFO:tasks.rados.rados.0.smithi097.stdout:275: writing smithi09731216-22 from 360448 to 1081344 tid 1 2024-04-18T21:52:45.555 INFO:tasks.rados.rados.0.smithi097.stdout:275: writing smithi09731216-22 from 1081344 to 1884160 tid 2 2024-04-18T21:52:45.559 INFO:tasks.rados.rados.0.smithi097.stdout:275: writing smithi09731216-22 from 1884160 to 2244608 tid 3 2024-04-18T21:52:45.559 INFO:tasks.rados.rados.0.smithi097.stdout:276: read oid 37 snap 21 2024-04-18T21:52:45.559 INFO:tasks.rados.rados.0.smithi097.stdout:276: expect (ObjNum 91 snap 18 seq_num 91) 2024-04-18T21:52:45.559 INFO:tasks.rados.rados.0.smithi097.stdout:277: snap_remove snap 22 2024-04-18T21:52:45.616 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:45.615+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:45.679 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:45.675+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:45.693 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:45.693 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:45.693 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:45.693 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:45.693 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:45.707 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:45.780 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:45.774+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:45.833 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:45.831+0000 7f8622e9b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:106 3.6s0 3:64491ca4:::smithi09731216-36:head [append 0~589824,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.784935779000001, 2024-04-18T21:52:45.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.39237912, 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 106 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088749+0000", 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088750+0000", 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3859999999999999e-06 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089602+0000", 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000852537 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089629+0000", 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6066000000000001e-05 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089648+0000", 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9449999999999998e-05 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174464+0000", 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816027000000002 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174469+0000", 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5970000000000002e-06 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481023+0000", 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3065548339999999 2024-04-18T21:52:45.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.481128+0000", 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010422299999999999 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:107 3.6s0 3:64491ca4:::smithi09731216-36:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.784006816000002, 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.301432395, 2024-04-18T21:52:45.881 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 107 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089678+0000", 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089679+0000", 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.686e-06 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089697+0000", 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7717e-05 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089714+0000", 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7433000000000001e-05 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089730+0000", 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5953000000000001e-05 2024-04-18T21:52:45.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174547+0000", 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084816707000000005 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174550+0000", 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9100000000000001e-06 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.983958+0000", 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8094078099999999 2024-04-18T21:52:45.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984087+0000", 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012884800000000001 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.984411+0000", 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032455500000000003 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986825+0000", 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024135609999999998 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391001+0000", 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40417611399999998 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391110+0000", 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109101 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:108 3.6s0 3:64491ca4:::smithi09731216-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.783933171000001, 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4199299170000002, 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 108 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.888 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089751+0000", 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089753+0000", 2024-04-18T21:52:45.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.46e-06 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089759+0000", 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5969999999999997e-06 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089782+0000", 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2546999999999999e-05 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089799+0000", 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6742999999999999e-05 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174653+0000", 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084854180000000001 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.886e-06 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391631+0000", 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2169758150000001 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.391728+0000", 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6861000000000007e-05 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509681+0000", 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.117952828 2024-04-18T21:52:45.892 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:109 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~606208,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.783859618000001, 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4179864210000002, 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.893 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089825+0000", 2024-04-18T21:52:45.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.089826+0000", 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.147e-06 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090694+0000", 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00086783200000000002 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090726+0000", 2024-04-18T21:52:45.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2379999999999998e-05 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090746+0000", 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9466999999999998e-05 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174599+0000", 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083853091000000005 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174600+0000", 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.097e-06 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985378+0000", 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.810778236 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.985469+0000", 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0805999999999998e-05 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.986312+0000", 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084315300000000002 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089266+0000", 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10295370099999999 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507611+0000", 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41834516199999999 2024-04-18T21:52:45.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.507811+0000", 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000200349 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:110 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.782916215, 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.429779607, 2024-04-18T21:52:45.899 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 110 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090768+0000", 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090769+0000", 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090782+0000", 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2547e-05 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090797+0000", 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5109000000000001e-05 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090807+0000", 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8670000000000006e-06 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174656+0000", 2024-04-18T21:52:45.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083849218000000003 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.174657+0000", 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8700000000000004e-07 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087432+0000", 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9127752569999998 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.087519+0000", 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6668000000000004e-05 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088034+0000", 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00051538599999999995 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.089552+0000", 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001518295 2024-04-18T21:52:45.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520465+0000", 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.43091233000000001 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520548+0000", 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2947000000000004e-05 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:111 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.782866476000002, 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5330603059999999, 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 111 2024-04-18T21:52:45.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090818+0000", 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090819+0000", 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6000000000000003e-07 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090822+0000", 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.066e-06 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090834+0000", 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2486999999999999e-05 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.090844+0000", 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9899999999999992e-06 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275049+0000", 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.184205065 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.275055+0000", 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4389999999999999e-06 2024-04-18T21:52:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088362+0000", 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8133068830000001 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088621+0000", 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025974499999999997 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520968+0000", 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.432346972 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521091+0000", 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122263 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.623878+0000", 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102787636 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.911 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:127 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~720896,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 57.059391491, 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50504046999999996, 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.912 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 127 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814293+0000", 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.814297+0000", 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3120000000000004e-06 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815282+0000", 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0009845469999999999 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815306+0000", 2024-04-18T21:52:45.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4488999999999999e-05 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815321+0000", 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4365999999999999e-05 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901001+0000", 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085680191000000003 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901126+0000", 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125506 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.902346+0000", 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012201989999999999 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904676+0000", 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023299459999999998 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319226+0000", 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.414549324 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.319333+0000", 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010759 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:128 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:45.917 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 57.058336734999997, 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70473296200000002, 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 128 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815348+0000", 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815349+0000", 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.003e-06 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815353+0000", 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5700000000000003e-06 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815362+0000", 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4540000000000007e-06 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.815369+0000", 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4059999999999999e-06 2024-04-18T21:52:45.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903371+0000", 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088001653999999999 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.903511+0000", 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000139889 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.904181+0000", 2024-04-18T21:52:45.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067014400000000001 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.905274+0000", 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010929430000000001 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519876+0000", 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61460184500000004 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.520081+0000", 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020505400000000001 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.1s0,GRANT,e:32,from:5(4),reservation_nonce:5)", 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.972150485, 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61973115499999998, 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.923 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901534+0000", 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901535+0000", 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.206e-06 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901541+0000", 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.331e-06 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.901559+0000", 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7977000000000001e-05 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.001690+0000", 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100130964 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521089+0000", 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51939875099999999 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521142+0000", 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3324e-05 2024-04-18T21:52:45.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521265+0000", 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122602 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:130 3.1s0 3:87c089d9:::smithi09731216-44:head [append 0~737280,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.962678216, 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61075046099999997, 2024-04-18T21:52:45.927 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 130 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911006+0000", 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911008+0000", 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.356e-06 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911859+0000", 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085008399999999997 2024-04-18T21:52:45.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911877+0000", 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8147000000000001e-05 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911886+0000", 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012129+0000", 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100243058 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.012391+0000", 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026173100000000002 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.014631+0000", 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002240344 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118257+0000", 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103625859 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521564+0000", 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40330724200000001 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521757+0000", 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019236000000000001 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:131 3.1s0 3:87c089d9:::smithi09731216-44:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.961779907, 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.70851564899999997, 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.933 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 131 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911904+0000", 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911905+0000", 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9800000000000002e-07 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911910+0000", 2024-04-18T21:52:45.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4499999999999997e-06 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911919+0000", 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2799999999999992e-06 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.911927+0000", 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5700000000000004e-06 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017049+0000", 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105122775 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017158+0000", 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010845600000000001 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.017640+0000", 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00048220899999999999 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.118261+0000", 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100621063 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620272+0000", 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50201088400000005 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.620420+0000", 2024-04-18T21:52:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000148064 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.766795465000001, 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31260053300000001, 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.939 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106889+0000", 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106890+0000", 2024-04-18T21:52:45.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8599999999999996e-07 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106896+0000", 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9139999999999996e-06 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106911+0000", 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4800999999999999e-05 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.106920+0000", 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9379999999999998e-06 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419470+0000", 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.312550204 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.419489+0000", 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.969e-05 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.942 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.11s0 33/31 ECSubWriteReply(tid=10, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.668870691999999, 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.316195273, 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204814+0000", 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1699999999999998e-07 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204818+0000", 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5659999999999998e-06 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204828+0000", 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0478e-05 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204836+0000", 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.413e-06 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519654+0000", 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.314818722 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.521009+0000", 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001354577 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=13, last_complete=32'5, committed=1, applied=1))", 2024-04-18T21:52:45.946 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.668837007, 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31487986099999998, 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204847+0000", 2024-04-18T21:52:45.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204848+0000", 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8299999999999998e-07 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204850+0000", 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6429999999999999e-06 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204859+0000", 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0550000000000005e-06 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204866+0000", 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5320000000000003e-06 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519669+0000", 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31480259199999999 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519727+0000", 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8656000000000002e-05 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 33/31 ECSubWriteReply(tid=14, last_complete=32'6, committed=1, applied=1))", 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 56.66880415, 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31508690700000003, 2024-04-18T21:52:45.950 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204880+0000", 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204881+0000", 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3000000000000002e-07 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204884+0000", 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.976e-06 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204900+0000", 2024-04-18T21:52:45.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6418000000000001e-05 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.204908+0000", 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3080000000000008e-06 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519923+0000", 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31501511199999999 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.519967+0000", 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3763000000000001e-05 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:524 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~770048] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.380459+0000", 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.493225765999998, 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.709551441, 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.954 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 524 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.380459+0000", 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.380459+0000", 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.380463+0000", 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0130000000000004e-06 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382388+0000", 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019258980000000001 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382426+0000", 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7521000000000002e-05 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382449+0000", 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3371999999999999e-05 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386170+0000", 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037207609999999999 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386450+0000", 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027958300000000002 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.390115+0000", 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036657510000000001 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.403183+0000", 2024-04-18T21:52:45.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.013067066 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.089832+0000", 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.68664989300000001 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.090010+0000", 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017758300000000001 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:525 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~475136] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.385002+0000", 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.488682780000001, 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.80581454200000002, 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 525 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.960 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.385002+0000", 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.385002+0000", 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.385004+0000", 2024-04-18T21:52:45.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8660000000000002e-06 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386191+0000", 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001186335 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386225+0000", 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4538e-05 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386248+0000", 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2402e-05 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.395454+0000", 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0092064380000000008 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.395619+0000", 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000165274 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.397339+0000", 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017190899999999999 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.404292+0000", 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069534590000000004 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.190633+0000", 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78634107099999995 2024-04-18T21:52:45.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.190816+0000", 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018306900000000001 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:526 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [append 0~294912] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.386285+0000", 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.487399218, 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.80528929800000004, 2024-04-18T21:52:45.965 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 526 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386285+0000", 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386285+0000", 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386288+0000", 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6240000000000002e-06 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386930+0000", 2024-04-18T21:52:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00064243500000000001 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386966+0000", 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5340999999999999e-05 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.386994+0000", 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8289e-05 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.398679+0000", 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01168482 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.398836+0000", 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000157355 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.400287+0000", 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014512450000000001 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.404323+0000", 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040361559999999999 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.191308+0000", 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78698441100000005 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.191574+0000", 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000266622 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.970 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:527 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [setxattr _header (57) in=7b] snapc 13=[13,12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.387028+0000", 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.486656307000001, 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90512298099999999, 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.971 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 527 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387028+0000", 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387028+0000", 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387030+0000", 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.866e-06 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387039+0000", 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9060000000000008e-06 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387060+0000", 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0979e-05 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387076+0000", 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6175999999999999e-05 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.401384+0000", 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.014307998000000001 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.401528+0000", 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014453400000000001 2024-04-18T21:52:45.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.402440+0000", 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00091111999999999996 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.404362+0000", 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001922656 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.291910+0000", 2024-04-18T21:52:45.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88754813200000005 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.292151+0000", 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000240614 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:528 3.16s0 3:6fc2bcb0:::smithi09731216-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.387098+0000", 2024-04-18T21:52:45.976 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.486586127999999, 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2983067880000001, 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 528 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:45.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387098+0000", 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387098+0000", 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387099+0000", 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.243e-06 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387106+0000", 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0589999999999999e-06 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387125+0000", 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9488999999999999e-05 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.387159+0000", 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4372000000000002e-05 2024-04-18T21:52:45.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.402967+0000", 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015808106999999998 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.403155+0000", 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018745599999999999 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:45.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.293393+0000", 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.89023824600000001 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.293571+0000", 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017742999999999999 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.685405+0000", 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39183438599999998 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.981 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:45.982 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:45.982 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:45.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:45.984 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:45.978+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:46.084 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:46.079+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:46.116 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:46.116 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:46.116 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:52:46.116 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:46.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:46.217 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:46.215+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:46.285 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:46.283+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:46.295 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:46.295 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:52:46.295 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:52:46.295 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:52:46.296 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:46.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:46.387 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:46.382+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:46.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:46.420 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:52:46.431 INFO:tasks.rados.rados.0.smithi097.stdout:277: done (8 left) 2024-04-18T21:52:46.431 INFO:tasks.rados.rados.0.smithi097.stdout:278: copy_from oid 2 from oid 7 current snap is 23 2024-04-18T21:52:46.431 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 17 current snap is 23 2024-04-18T21:52:46.431 INFO:tasks.rados.rados.0.smithi097.stdout:279: seq_num 113 ranges {704512=737280,1441792=557056,1998848=606208} 2024-04-18T21:52:46.439 INFO:tasks.rados.rados.0.smithi097.stdout:279: writing smithi09731216-17 from 704512 to 1441792 tid 1 2024-04-18T21:52:46.443 INFO:tasks.rados.rados.0.smithi097.stdout:279: writing smithi09731216-17 from 1441792 to 1998848 tid 2 2024-04-18T21:52:46.449 INFO:tasks.rados.rados.0.smithi097.stdout:279: writing smithi09731216-17 from 1998848 to 2605056 tid 3 2024-04-18T21:52:46.449 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 50 current snap is 23 2024-04-18T21:52:46.449 INFO:tasks.rados.rados.0.smithi097.stdout:280: seq_num 114 ranges {1851392=753664,2605056=688128,3293184=475136} 2024-04-18T21:52:46.467 INFO:tasks.rados.rados.0.smithi097.stdout:280: writing smithi09731216-50 from 1851392 to 2605056 tid 1 2024-04-18T21:52:46.472 INFO:tasks.rados.rados.0.smithi097.stdout:280: writing smithi09731216-50 from 2605056 to 3293184 tid 2 2024-04-18T21:52:46.476 INFO:tasks.rados.rados.0.smithi097.stdout:280: writing smithi09731216-50 from 3293184 to 3768320 tid 3 2024-04-18T21:52:46.476 INFO:tasks.rados.rados.0.smithi097.stdout:281: read oid 45 snap -1 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:281: expect (ObjNum 84 snap 14 seq_num 84) 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:282: read oid 8 snap 21 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:282: expect deleted 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:283: setattr oid 11 current snap is 23 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:275: finishing write tid 1 to smithi09731216-22 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:275: finishing write tid 2 to smithi09731216-22 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:275: finishing write tid 3 to smithi09731216-22 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:275: finishing write tid 4 to smithi09731216-22 2024-04-18T21:52:46.477 INFO:tasks.rados.rados.0.smithi097.stdout:265: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-18T21:52:46.482 INFO:tasks.rados.rados.0.smithi097.stdout:275: finishing write tid 5 to smithi09731216-22 2024-04-18T21:52:46.482 INFO:tasks.rados.rados.0.smithi097.stdout:275: oid 22 updating version 0 to 137 2024-04-18T21:52:46.482 INFO:tasks.rados.rados.0.smithi097.stdout:275: oid 22 updating version 137 to 139 2024-04-18T21:52:46.482 INFO:tasks.rados.rados.0.smithi097.stdout:275: oid 22 updating version 139 to 140 2024-04-18T21:52:46.482 INFO:tasks.rados.rados.0.smithi097.stdout:275: oid 22 version 140 is already newer than 138 2024-04-18T21:52:46.482 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 22 v 140 (ObjNum 112 snap 23 seq_num 112) dirty exists 2024-04-18T21:52:46.482 INFO:tasks.rados.rados.0.smithi097.stdout:275: left oid 22 (ObjNum 112 snap 23 seq_num 112) 2024-04-18T21:52:46.483 INFO:tasks.rados.rados.0.smithi097.stdout:265: done (13 left) 2024-04-18T21:52:46.483 INFO:tasks.rados.rados.0.smithi097.stdout:272: done (12 left) 2024-04-18T21:52:46.483 INFO:tasks.rados.rados.0.smithi097.stdout:275: done (11 left) 2024-04-18T21:52:46.483 INFO:tasks.rados.rados.0.smithi097.stdout:284: rollback oid 42 current snap is 23 2024-04-18T21:52:46.483 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 42 to 21 2024-04-18T21:52:46.483 INFO:tasks.rados.rados.0.smithi097.stdout:285: delete oid 39 current snap is 23 2024-04-18T21:52:46.487 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:46.483+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:46.504 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:46.504 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:52:46.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:52:46.504 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:52:46.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.504 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:155 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'109, mlcod=24'105)", 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 65.197173336999995, 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30724263200000002, 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306493+0000", 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6800000000000002e-07 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306525+0000", 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2228999999999999e-05 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306532+0000", 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7469999999999999e-06 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306539+0000", 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1250000000000004e-06 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.307270+0000", 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00073067900000000003 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.307309+0000", 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9830000000000003e-05 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613662+0000", 2024-04-18T21:52:46.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30635288799999999 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613735+0000", 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2866000000000003e-05 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:156 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'110, mlcod=24'105)", 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 65.197120564000002, 2024-04-18T21:52:46.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30725738200000002, 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306545+0000", 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306546+0000", 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.16e-07 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306548+0000", 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.672e-06 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306553+0000", 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0730000000000004e-06 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306556+0000", 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.959e-06 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408193+0000", 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101636647 2024-04-18T21:52:46.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408269+0000", 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5767999999999997e-05 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613742+0000", 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.205473081 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613803+0000", 2024-04-18T21:52:46.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0965999999999999e-05 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.4165.0:157 1.0 e25/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'111, mlcod=24'105)", 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 65.197105601000004, 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30730748800000002, 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306560+0000", 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.72e-07 2024-04-18T21:52:46.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306563+0000", 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2730000000000001e-06 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306572+0000", 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2390000000000008e-06 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.306577+0000", 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7709999999999997e-06 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408635+0000", 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102058725 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.408654+0000", 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8921000000000001e-05 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613808+0000", 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20515377100000001 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:41.613868+0000", 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9616e-05 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.518 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.141188853999999, 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40570043900000002, 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362477+0000", 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362478+0000", 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4279999999999999e-06 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362656+0000", 2024-04-18T21:52:46.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017710100000000001 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.362673+0000", 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7802e-05 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462745+0000", 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100071484 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.767971+0000", 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30522610500000003 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.768177+0000", 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020651900000000001 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.522 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.074299617999998, 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44082012100000001, 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429366+0000", 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429367+0000", 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5300000000000004e-07 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429643+0000", 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027611500000000001 2024-04-18T21:52:46.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.429665+0000", 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1603e-05 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.529793+0000", 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100128809 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868616+0000", 2024-04-18T21:52:46.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.33882269300000001 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870186+0000", 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015703480000000001 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:46.526 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.040829746999997, 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40574496100000002, 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:46.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462836+0000", 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0100000000000001e-07 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462933+0000", 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6247000000000002e-05 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462943+0000", 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.944e-06 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.462948+0000", 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4979999999999997e-06 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868491+0000", 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40554244699999997 2024-04-18T21:52:46.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.868581+0000", 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0524000000000003e-05 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.037919342000002, 2024-04-18T21:52:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71932642099999999, 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465747+0000", 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9599999999999998e-07 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465774+0000", 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6846e-05 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465781+0000", 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2640000000000002e-06 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465785+0000", 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7759999999999999e-06 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.185006+0000", 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71922088500000003 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.185073+0000", 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6853999999999994e-05 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.037850329999998, 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82169303100000002, 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465816+0000", 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4000000000000002e-07 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465848+0000", 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1609000000000002e-05 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465853+0000", 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.366e-06 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465856+0000", 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.151e-06 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286522+0000", 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.82066616299999995 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.287509+0000", 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0009863020000000001 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 49.037753524999999, 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40453295099999997, 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465912+0000", 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465913+0000", 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6199999999999998e-07 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.465996+0000", 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3168000000000001e-05 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.466007+0000", 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0662000000000001e-05 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.466014+0000", 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8639999999999998e-06 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870275+0000", 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.404261702 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.870445+0000", 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016999300000000001 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.542 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.972285425999999, 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.75509254699999995, 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531380+0000", 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531387+0000", 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2899999999999999e-06 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531557+0000", 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016995899999999999 2024-04-18T21:52:46.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531588+0000", 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1069999999999999e-05 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.531606+0000", 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8479999999999999e-05 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286385+0000", 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.754778964 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.286473+0000", 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7783999999999998e-05 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.546 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 40/39 ECSubReadReply(tid=1, attrs_read=0))", 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 48.937575799000001, 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858464, 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566090+0000", 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566092+0000", 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5799999999999999e-06 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566247+0000", 2024-04-18T21:52:46.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155613 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566277+0000", 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0111999999999999e-05 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.566315+0000", 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7141999999999998e-05 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.974376+0000", 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.408061795 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:57.974675+0000", 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029839800000000001 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=7, attrs_read=0))", 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.496898866000002, 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41262958999999999, 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006767+0000", 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006768+0000", 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4139999999999999e-06 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006798+0000", 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9499e-05 2024-04-18T21:52:46.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006823+0000", 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4706999999999999e-05 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.006842+0000", 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9507000000000001e-05 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418445+0000", 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41160240300000001 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419397+0000", 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095206000000000002 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpRead(3.10s1 41/39 ECSubRead(tid=11, to_read={3:0e754656:::smithi09731216-19:head=0,4096,0}, subchunks={3:0e754656:::smithi09731216-19:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:46.554 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.482075262000002, 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39881851499999998, 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021591+0000", 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021616+0000", 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5575000000000002e-05 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.021627+0000", 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1099e-05 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.420045+0000", 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39841796699999998 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.420409+0000", 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036387399999999998 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:46.558 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.481424273000002, 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39759610699999998, 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022242+0000", 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6199999999999997e-07 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022253+0000", 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0283e-05 2024-04-18T21:52:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022265+0000", 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2302e-05 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022271+0000", 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2550000000000003e-06 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419761+0000", 2024-04-18T21:52:46.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39748950500000002 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419838+0000", 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7000000000000001e-05 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:46.562 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.481200501000004, 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39718676600000002, 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022465+0000", 2024-04-18T21:52:46.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022467+0000", 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3030000000000001e-06 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022485+0000", 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8363999999999999e-05 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022504+0000", 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9283000000000002e-05 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022514+0000", 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8430000000000005e-06 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419513+0000", 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39699880100000001 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419652+0000", 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013917199999999999 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.481081625000002, 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39738200800000001, 2024-04-18T21:52:46.566 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022584+0000", 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022585+0000", 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7599999999999996e-07 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022601+0000", 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6203000000000001e-05 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022613+0000", 2024-04-18T21:52:46.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1334e-05 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022618+0000", 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4110000000000004e-06 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419896+0000", 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39727772700000002 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.419966+0000", 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0556999999999995e-05 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.10s1 41/39 ECSubWriteReply(tid=10, last_complete=41'16, committed=1, applied=1))", 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.481029028999998, 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39534124999999998, 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.570 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022637+0000", 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022640+0000", 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0029999999999999e-06 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022652+0000", 2024-04-18T21:52:46.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1585e-05 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.022656+0000", 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5739999999999997e-06 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.417932+0000", 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39527527800000001 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.417978+0000", 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6350000000000002e-05 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.10s1 41/39 ECSubWriteReply(tid=10, last_complete=41'16, committed=1, applied=1))", 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.398968730999997, 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31366916900000003, 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.574 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104697+0000", 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104699+0000", 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.686e-06 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104713+0000", 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4296e-05 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104742+0000", 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9204999999999999e-05 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.104758+0000", 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5254e-05 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418210+0000", 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31345283600000001 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418366+0000", 2024-04-18T21:52:46.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155892 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "replica_scrub(pg: 3.18s6,from:0'0,to:38'13,epoch:41/31,start:3:18000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.397540206999999, 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31261068600000003, 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:52:46.578 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106126+0000", 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106127+0000", 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5000000000000001e-07 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106133+0000", 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8940000000000004e-06 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106149+0000", 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5534000000000001e-05 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.106159+0000", 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.047e-05 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418620+0000", 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31246021499999999 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418658+0000", 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8566000000000003e-05 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.418736+0000", 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8157000000000004e-05 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpReadReply(3.10s1 41/39 ECSubReadReply(tid=11, attrs_read=0))", 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.381975103999999, 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39824586299999998, 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:52:46.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121691+0000", 2024-04-18T21:52:46.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:52:46.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:52:46.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121692+0000", 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5499999999999997e-07 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121705+0000", 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3207000000000001e-05 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121718+0000", 2024-04-18T21:52:46.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2913e-05 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.121724+0000", 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5030000000000002e-06 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.519849+0000", 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39812503900000001 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.519937+0000", 2024-04-18T21:52:46.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7346e-05 2024-04-18T21:52:46.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.588 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:46.588 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:46.588 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:46.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:46.590 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:46.591 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:46.591 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:46.587+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:46.638 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:46.634+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:46.640 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:46.635+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:46.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:46.688 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:46.690 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:46.687+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:46.709 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:46.709 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:46.734 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:46.734 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:46.741 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 11 v 118 (ObjNum 106 snap 19 seq_num 106) dirty exists 2024-04-18T21:52:46.759 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:46.759 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:46.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:46.817 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:46.815+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:46.846 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:46.847 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:46.871 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:46.872 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:46.876 INFO:tasks.rados.rados.0.smithi097.stdout:279: finishing write tid 1 to smithi09731216-17 2024-04-18T21:52:46.877 INFO:tasks.rados.rados.0.smithi097.stdout:279: finishing write tid 2 to smithi09731216-17 2024-04-18T21:52:46.878 INFO:tasks.rados.rados.0.smithi097.stdout:279: finishing write tid 3 to smithi09731216-17 2024-04-18T21:52:46.879 INFO:tasks.rados.rados.0.smithi097.stdout:279: finishing write tid 4 to smithi09731216-17 2024-04-18T21:52:46.892 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:46.890+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:46.896 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:46.896 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:46.926 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:46.926 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:46.940 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:46.940 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:46.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:46.955 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:46.955 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:46.961 INFO:tasks.rados.rados.0.smithi097.stdout:281: expect (ObjNum 84 snap 14 seq_num 84) 2024-04-18T21:52:46.982 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:46.982 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:46.994 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:46.991+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:47.009 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:47.009 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:47.086 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:47.086 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:47.095 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:47.090+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:47.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:47.181 INFO:tasks.rados.rados.0.smithi097.stdout:279: finishing write tid 5 to smithi09731216-17 2024-04-18T21:52:47.181 INFO:tasks.rados.rados.0.smithi097.stdout:279: oid 17 updating version 0 to 142 2024-04-18T21:52:47.181 INFO:tasks.rados.rados.0.smithi097.stdout:279: oid 17 updating version 142 to 145 2024-04-18T21:52:47.181 INFO:tasks.rados.rados.0.smithi097.stdout:279: oid 17 version 145 is already newer than 143 2024-04-18T21:52:47.181 INFO:tasks.rados.rados.0.smithi097.stdout:279: oid 17 version 145 is already newer than 144 2024-04-18T21:52:47.181 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 17 v 145 (ObjNum 113 snap 23 seq_num 113) dirty exists 2024-04-18T21:52:47.181 INFO:tasks.rados.rados.0.smithi097.stdout:279: left oid 17 (ObjNum 113 snap 23 seq_num 113) 2024-04-18T21:52:47.196 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.190+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:47.225 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:47.225 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:47.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:47.297 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:47.295+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:47.362 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:47.363 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:47.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:47.421 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.419+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:47.431 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.431 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.431 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.431 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.431 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.431 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.431 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.431 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.432 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.433 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.434 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.435 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.436 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.437 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.439 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.440 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.441 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.442 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.443 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.444 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.445 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.446 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.446 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.446 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.446 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.446 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.446 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.446 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.447 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.448 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.449 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.450 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.451 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.452 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.423+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.453 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.454 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.455 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.455 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.455 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.455 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.455 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.455 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.455 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:47.427+0000 7f861ee93640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.456 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.426+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.456 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.430+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.456 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.438+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.456 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.438+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.456 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.442+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.486 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.482+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.499 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:47.495+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:47.541 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.538+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.549 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:47.549 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:47.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:47.600 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:47.595+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:47.701 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:47.698+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:47.728 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:47.728 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:47.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:47.801 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:47.798+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:47.861 INFO:tasks.rados.rados.0.smithi097.stdout:279: done (12 left) 2024-04-18T21:52:47.861 INFO:tasks.rados.rados.0.smithi097.stdout:281: done (11 left) 2024-04-18T21:52:47.861 INFO:tasks.rados.rados.0.smithi097.stdout:283: done (10 left) 2024-04-18T21:52:47.861 INFO:tasks.rados.rados.0.smithi097.stdout:285: done (9 left) 2024-04-18T21:52:47.861 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 6 current snap is 23 2024-04-18T21:52:47.861 INFO:tasks.rados.rados.0.smithi097.stdout:286: seq_num 116 ranges {1474560=786432,2260992=458752,2719744=442368,3162112=262144} 2024-04-18T21:52:47.875 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:47.875 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:47.876 INFO:tasks.rados.rados.0.smithi097.stdout:286: writing smithi09731216-6 from 1474560 to 2260992 tid 1 2024-04-18T21:52:47.878 INFO:tasks.rados.rados.0.smithi097.stdout:286: writing smithi09731216-6 from 2260992 to 2719744 tid 2 2024-04-18T21:52:47.881 INFO:tasks.rados.rados.0.smithi097.stdout:286: writing smithi09731216-6 from 2719744 to 3162112 tid 3 2024-04-18T21:52:47.882 INFO:tasks.rados.rados.0.smithi097.stdout:286: writing smithi09731216-6 from 3162112 to 3424256 tid 4 2024-04-18T21:52:47.883 INFO:tasks.rados.rados.0.smithi097.stdout:287: setattr oid 26 current snap is 23 2024-04-18T21:52:47.883 INFO:tasks.rados.rados.0.smithi097.stdout:278: finishing copy_from racing read to smithi09731216-2 2024-04-18T21:52:47.883 INFO:tasks.rados.rados.0.smithi097.stdout:288: setattr oid 11 current snap is 23 2024-04-18T21:52:47.884 INFO:tasks.rados.rados.0.smithi097.stdout:289: snap_remove snap 19 2024-04-18T21:52:47.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:47.902 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:47.898+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:47.974 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.970+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:47.975 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:47.970+0000 7f7106ca9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:48.002 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:47.998+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:48.009 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:48.009 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:48.023 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:48.078 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:48.074+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:48.078 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:48.074+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:48.103 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:48.098+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:48.157 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:48.157 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:48.174 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:48.181 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:48.178+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:48.181 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:48.178+0000 7f710acb1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:48.225 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:48.222+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:48.304 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:48.302+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:48.306 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:48.306 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:48.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:48.405 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:48.402+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:48.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-18T21:52:48.470 INFO:tasks.rados.rados.0.smithi097.stdout:289: done (12 left) 2024-04-18T21:52:48.470 INFO:tasks.rados.rados.0.smithi097.stdout:290: read oid 22 snap -1 2024-04-18T21:52:48.470 INFO:tasks.rados.rados.0.smithi097.stdout:290: expect (ObjNum 112 snap 23 seq_num 112) 2024-04-18T21:52:48.470 INFO:tasks.rados.rados.0.smithi097.stdout:291: read oid 17 snap -1 2024-04-18T21:52:48.470 INFO:tasks.rados.rados.0.smithi097.stdout:291: expect (ObjNum 113 snap 23 seq_num 113) 2024-04-18T21:52:48.470 INFO:tasks.rados.rados.0.smithi097.stdout:292: read oid 45 snap -1 2024-04-18T21:52:48.470 INFO:tasks.rados.rados.0.smithi097.stdout:292: expect (ObjNum 84 snap 14 seq_num 84) 2024-04-18T21:52:48.470 INFO:tasks.rados.rados.0.smithi097.stdout:293: snap_remove snap 8 2024-04-18T21:52:48.471 INFO:tasks.rados.rados.0.smithi097.stdout:266: finishing write tid 3 to smithi09731216-12 2024-04-18T21:52:48.471 INFO:tasks.rados.rados.0.smithi097.stdout:266: finishing write tid 4 to smithi09731216-12 2024-04-18T21:52:48.493 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:48.493 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:48.506 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:48.502+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:48.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:48.607 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:48.602+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:48.673 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:48.673 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:48.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:48.733 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:48.730+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:48.809 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:48.806+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:48.858 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:48.858 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:48.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:48.910 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:48.906+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:48.997 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:48.998 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:49.011 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:49.007+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:49.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:49.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:49.112 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:49.106+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:49.137 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:49.137 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:49.151 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:49.187 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:49.187 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:49.205 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:49.205 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:49.213 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:49.210+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:49.221 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:49.222 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:49.238 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:49.238 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:49.256 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:49.257 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:49.277 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:49.277 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:49.314 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:49.310+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:49.320 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:49.320 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:49.334 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:49.393 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:49.393 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:49.414 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:49.411+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:49.419 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:49.420 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:49.449 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:49.449 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:239 3.as0 3:5bb4af71:::smithi09731216-7:head [delete] snapc 4=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 55.446380169999998, 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40857630099999998, 2024-04-18T21:52:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014034+0000", 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2500000000000001e-06 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014044+0000", 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1889999999999996e-06 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014067+0000", 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3251000000000002e-05 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114145+0000", 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078193 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114306+0000", 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161044 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114320+0000", 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4044e-05 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317101+0000", 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20278075300000001 2024-04-18T21:52:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317362+0000", 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026128799999999998 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.319467+0000", 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021048719999999998 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.326280+0000", 2024-04-18T21:52:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006813375 2024-04-18T21:52:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422400+0000", 2024-04-18T21:52:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096119880000000005 2024-04-18T21:52:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422608+0000", 2024-04-18T21:52:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020816199999999999 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:308 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~737280] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 51.369940094, 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69365002799999997, 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 308 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090475+0000", 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.976e-06 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091768+0000", 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00129317 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091803+0000", 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4502999999999998e-05 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091826+0000", 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3235999999999999e-05 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270501+0000", 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.178674888 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270680+0000", 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017917599999999999 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.273556+0000", 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028757610000000001 2024-04-18T21:52:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281580+0000", 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080240799999999994 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783886+0000", 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50230573000000001 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784122+0000", 2024-04-18T21:52:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023650799999999999 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:309 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 51.367859101000001, 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69216249600000002, 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 309 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092555+0000", 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093122+0000", 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056744499999999997 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093146+0000", 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128999999999999e-05 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093165+0000", 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8300000000000001e-05 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275352+0000", 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18218714899999999 2024-04-18T21:52:49.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275632+0000", 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028059100000000001 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.277549+0000", 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001917087 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281584+0000", 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004034513 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784502+0000", 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50291762600000001 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784716+0000", 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021415599999999999 2024-04-18T21:52:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:310 3.as0 3:5bb4af71:::smithi09731216-7:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 51.367217930999999, 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69187687499999995, 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 310 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093196+0000", 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093210+0000", 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3517e-05 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093227+0000", 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7050000000000001e-05 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093243+0000", 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5965999999999999e-05 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279008+0000", 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.185765398 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279287+0000", 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027941400000000002 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.280700+0000", 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412581 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482172+0000", 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20147153900000001 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784956+0000", 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30278430299999998 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785071+0000", 2024-04-18T21:52:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115454 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:311 3.as0 3:5bb4af71:::smithi09731216-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 51.367142979, 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99553992999999996, 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 311 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093271+0000", 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093278+0000", 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1459999999999999e-06 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093295+0000", 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7408999999999999e-05 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093328+0000", 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2276000000000001e-05 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785901+0000", 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.692573628 2024-04-18T21:52:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.786181+0000", 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027934999999999999 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.088809+0000", 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30262875700000003 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:429 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.014736544000002, 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51405278700000001, 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 429 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:49.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445677+0000", 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5260000000000001e-06 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445685+0000", 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445703+0000", 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8295999999999999e-05 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445718+0000", 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5383000000000002e-05 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545773+0000", 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100054752 2024-04-18T21:52:49.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545785+0000", 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1783000000000001e-05 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557166+0000", 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011381578 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557443+0000", 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027702800000000002 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.558675+0000", 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012320479999999999 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659884+0000", 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101208162 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959511+0000", 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29962765699999999 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959729+0000", 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021733700000000001 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:495 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~737280] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:49.501 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 28.953373111000001, 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.664354899, 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 495 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507043+0000", 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.596e-06 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508389+0000", 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013461460000000001 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508432+0000", 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3294999999999998e-05 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508456+0000", 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4134e-05 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581521+0000", 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.073064403 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581827+0000", 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030644999999999999 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.586475+0000", 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046472989999999997 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.699953+0000", 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.113478523 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171243+0000", 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47128973499999999 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171394+0000", 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000151318 2024-04-18T21:52:49.506 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:496 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~573440] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 28.950185458, 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76183363900000001, 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.507 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 496 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510229+0000", 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.007e-06 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511015+0000", 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000785792 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511046+0000", 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0873000000000001e-05 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511061+0000", 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5293000000000001e-05 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.691993+0000", 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.180931701 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.692191+0000", 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019870099999999999 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.694402+0000", 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022106690000000002 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.701975+0000", 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075735730000000001 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.271924+0000", 2024-04-18T21:52:49.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56994823500000003 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272061+0000", 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013679499999999999 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:497 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~196608] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 28.949136096, 2024-04-18T21:52:49.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76114050600000005, 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 497 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:49.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511278+0000", 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.387e-06 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511608+0000", 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033003500000000001 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511637+0000", 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9706e-05 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511661+0000", 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3439999999999999e-05 2024-04-18T21:52:49.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696155+0000", 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18449448800000001 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696457+0000", 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000301496 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.698580+0000", 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021231879999999998 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.702633+0000", 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040534079999999997 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272305+0000", 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56967170199999995 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272417+0000", 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111656 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:535 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~802816] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:49.518 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.061927383, 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60500271100000003, 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 535 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:49.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398487+0000", 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0650000000000001e-06 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399399+0000", 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091160899999999997 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399417+0000", 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7915999999999999e-05 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399426+0000", 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7070000000000004e-06 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.485946+0000", 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086519677000000003 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.486140+0000", 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019389 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.488798+0000", 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002657707 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.600548+0000", 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11175003 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003305+0000", 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40275720799999998 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003488+0000", 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000182902 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.523 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:536 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~638976] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.056949725999999, 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60040761799999998, 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 536 2024-04-18T21:52:49.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:49.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.09e-07 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404080+0000", 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00061689199999999996 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404092+0000", 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2055000000000001e-05 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404101+0000", 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5650000000000002e-06 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491527+0000", 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087426312000000006 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491697+0000", 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017024799999999999 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.493862+0000", 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021641299999999998 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700761+0000", 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20689907199999999 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003718+0000", 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30295725099999998 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003870+0000", 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015228400000000001 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:537 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~131072] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.055980065, 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59990886700000001, 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.529 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 537 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404433+0000", 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.44e-07 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404610+0000", 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017705100000000001 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404620+0000", 2024-04-18T21:52:49.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8050000000000001e-06 2024-04-18T21:52:49.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404625+0000", 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5029999999999997e-06 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495333+0000", 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.090708141000000006 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495532+0000", 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001983 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.496836+0000", 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013041789999999999 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700772+0000", 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20393593900000001 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004176+0000", 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30340381399999999 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004341+0000", 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016549100000000001 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:538 3.10s0 3:0e754656:::smithi09731216-19:head [setxattr _header (57) in=7b] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.055778077999999, 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70001739100000004, 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 538 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404635+0000", 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6699999999999999e-07 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404637+0000", 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.683e-06 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404642+0000", 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.967e-06 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404647+0000", 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.25e-06 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.497929+0000", 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.093281966999999993 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498095+0000", 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016633800000000001 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498905+0000", 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081039199999999995 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700775+0000", 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20186959700000001 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104337+0000", 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40356159400000002 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104652+0000", 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031513599999999999 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:539 3.10s0 3:0e754656:::smithi09731216-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.055760004, 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90384237700000003, 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.541 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 539 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404653+0000", 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0600000000000001e-07 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404655+0000", 2024-04-18T21:52:49.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.95e-06 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404662+0000", 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9530000000000002e-06 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404666+0000", 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4089999999999996e-06 2024-04-18T21:52:49.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499383+0000", 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094717379000000004 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499540+0000", 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015672099999999999 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.105824+0000", 2024-04-18T21:52:49.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60628387699999997 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.106088+0000", 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026422700000000002 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.308495+0000", 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20240645500000001 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.546 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:572 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~507904 in=507904b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.927190869, 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60481051399999997, 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.547 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 572 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533224+0000", 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.824e-06 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534169+0000", 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094509200000000002 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534205+0000", 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5482999999999999e-05 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534228+0000", 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2806000000000002e-05 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634530+0000", 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10030222799999999 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634798+0000", 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026831400000000002 2024-04-18T21:52:49.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.638371+0000", 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035731249999999999 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744705+0000", 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106333831 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.137882+0000", 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39317701599999999 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.138032+0000", 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014979500000000001 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:573 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~606208 in=606208b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.926149560999999, 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70430669099999998, 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 573 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534265+0000", 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9800000000000001e-06 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535350+0000", 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010856869999999999 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535392+0000", 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1192999999999997e-05 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535427+0000", 2024-04-18T21:52:49.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5559999999999998e-05 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642525+0000", 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107097461 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642617+0000", 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2181000000000004e-05 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.643719+0000", 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011017449999999999 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747184+0000", 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103465025 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238444+0000", 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49126059700000002 2024-04-18T21:52:49.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238569+0000", 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012526200000000001 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:574 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~557056 in=557056b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.924954621000001, 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70342813999999998, 2024-04-18T21:52:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 574 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535460+0000", 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.796e-06 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540737+0000", 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052778449999999998 2024-04-18T21:52:49.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540774+0000", 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6372999999999999e-05 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540799+0000", 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5743999999999999e-05 2024-04-18T21:52:49.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744797+0000", 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20399719899999999 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744963+0000", 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016631799999999999 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.746246+0000", 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012830319999999999 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747841+0000", 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001595352 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238775+0000", 2024-04-18T21:52:49.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49093394899999998 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238886+0000", 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110532 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:575 3.as0 3:561bcae7:::smithi09731216-2:head [setxattr _header (56) in=63b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:49.564 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.919574168, 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69838045999999998, 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 575 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:49.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540841+0000", 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4540000000000001e-06 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540850+0000", 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2329999999999995e-06 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540874+0000", 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3980000000000001e-05 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540893+0000", 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9173000000000001e-05 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748051+0000", 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20715831100000001 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748150+0000", 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9155999999999997e-05 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748574+0000", 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042389100000000002 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.753028+0000", 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044531889999999998 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.239064+0000", 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.48603637100000002 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.239219+0000", 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154702 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.569 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 82/78 [PushReplyOp(3:0e754656:::smithi09731216-19:10)])", 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.165569898999999, 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81294819200000001, 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294850+0000", 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6299999999999998e-06 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294862+0000", 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.186e-05 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294892+0000", 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0103e-05 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294914+0000", 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1633000000000001e-05 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.107740+0000", 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81282626999999996 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.107791+0000", 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0695999999999997e-05 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.573 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 82/78 [PushReplyOp(3:0e754656:::smithi09731216-19:10)])", 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.164690755000001, 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91227883099999996, 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:52:49.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295723+0000", 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.703e-06 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295734+0000", 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.064e-05 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295762+0000", 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7606999999999998e-05 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295776+0000", 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.402e-05 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.207972+0000", 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91219607400000002 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.208000+0000", 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8787000000000001e-05 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.577 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:49.578 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is unset 2024-04-18T21:52:49.579 INFO:tasks.rados.rados.0.smithi097.stdout:293: done (15 left) 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:294: rollback oid 20 current snap is 23 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 20 to 21 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:278: finishing copy_from to smithi09731216-2 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 2 v 13 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:273: finishing write tid 1 to smithi09731216-49 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:273: finishing write tid 2 to smithi09731216-49 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:271: finishing rollback tid 0 to smithi09731216-33 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:273: finishing write tid 3 to smithi09731216-49 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:273: finishing write tid 4 to smithi09731216-49 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:286: finishing write tid 1 to smithi09731216-6 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:286: finishing write tid 2 to smithi09731216-6 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:266: finishing write tid 5 to smithi09731216-12 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:266: oid 12 updating version 0 to 62 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:266: oid 12 version 62 is already newer than 60 2024-04-18T21:52:49.580 INFO:tasks.rados.rados.0.smithi097.stdout:266: oid 12 version 62 is already newer than 61 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:266: oid 12 version 62 is already newer than 59 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 12 v 62 (ObjNum 110 snap 23 seq_num 110) dirty exists 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:266: left oid 12 (ObjNum 110 snap 23 seq_num 110) 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 11 v 119 (ObjNum 106 snap 19 seq_num 106) dirty exists 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:273: finishing write tid 5 to smithi09731216-49 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:273: oid 49 updating version 0 to 19 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:273: oid 49 version 19 is already newer than 17 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:273: oid 49 version 19 is already newer than 16 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:273: oid 49 version 19 is already newer than 18 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 49 v 19 (ObjNum 111 snap 23 seq_num 111) dirty exists 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:273: left oid 49 (ObjNum 111 snap 23 seq_num 111) 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:286: finishing write tid 3 to smithi09731216-6 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:284: finishing rollback tid 0 to smithi09731216-42 2024-04-18T21:52:49.581 INFO:tasks.rados.rados.0.smithi097.stdout:284: finishing rollback tid 1 to smithi09731216-42 2024-04-18T21:52:49.582 INFO:tasks.rados.rados.0.smithi097.stdout:284: finishing rollback tid 2 to smithi09731216-42 2024-04-18T21:52:49.582 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 42 v 16 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-18T21:52:49.582 INFO:tasks.rados.rados.0.smithi097.stdout:276: expect (ObjNum 91 snap 18 seq_num 91) 2024-04-18T21:52:49.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:49.583 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:49.574+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:49.583 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:49.583 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:49.584 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 30 v 35 (ObjNum 76 snap 10 seq_num 76) dirty exists 2024-04-18T21:52:49.584 INFO:tasks.rados.rados.0.smithi097.stdout:286: finishing write tid 4 to smithi09731216-6 2024-04-18T21:52:49.584 INFO:tasks.rados.rados.0.smithi097.stdout:286: finishing write tid 5 to smithi09731216-6 2024-04-18T21:52:49.584 INFO:tasks.rados.rados.0.smithi097.stdout:280: finishing write tid 1 to smithi09731216-50 2024-04-18T21:52:49.584 INFO:tasks.rados.rados.0.smithi097.stdout:280: finishing write tid 2 to smithi09731216-50 2024-04-18T21:52:49.584 INFO:tasks.rados.rados.0.smithi097.stdout:280: finishing write tid 3 to smithi09731216-50 2024-04-18T21:52:49.584 INFO:tasks.rados.rados.0.smithi097.stdout:280: finishing write tid 4 to smithi09731216-50 2024-04-18T21:52:49.585 INFO:tasks.rados.rados.0.smithi097.stdout:292: expect (ObjNum 84 snap 14 seq_num 84) 2024-04-18T21:52:49.595 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 26 v 11 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-18T21:52:49.595 INFO:tasks.rados.rados.0.smithi097.stdout:271: finishing rollback tid 1 to smithi09731216-33 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:271: finishing rollback tid 2 to smithi09731216-33 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:286: finishing write tid 6 to smithi09731216-6 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:286: oid 6 updating version 0 to 148 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:286: oid 6 version 148 is already newer than 147 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:286: oid 6 updating version 148 to 150 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:286: oid 6 updating version 150 to 151 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:286: oid 6 version 151 is already newer than 149 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 6 v 151 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:286: left oid 6 (ObjNum 116 snap 23 seq_num 116) 2024-04-18T21:52:49.596 INFO:tasks.rados.rados.0.smithi097.stdout:280: finishing write tid 5 to smithi09731216-50 2024-04-18T21:52:49.597 INFO:tasks.rados.rados.0.smithi097.stdout:280: oid 50 updating version 0 to 32 2024-04-18T21:52:49.597 INFO:tasks.rados.rados.0.smithi097.stdout:280: oid 50 version 32 is already newer than 31 2024-04-18T21:52:49.597 INFO:tasks.rados.rados.0.smithi097.stdout:280: oid 50 updating version 32 to 33 2024-04-18T21:52:49.597 INFO:tasks.rados.rados.0.smithi097.stdout:280: oid 50 updating version 33 to 34 2024-04-18T21:52:49.597 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 50 v 34 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-04-18T21:52:49.597 INFO:tasks.rados.rados.0.smithi097.stdout:280: left oid 50 (ObjNum 114 snap 23 seq_num 114) 2024-04-18T21:52:49.597 INFO:tasks.rados.rados.0.smithi097.stdout:290: expect (ObjNum 112 snap 23 seq_num 112) 2024-04-18T21:52:49.608 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:49.608 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:49.635 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:49.661 INFO:tasks.rados.rados.0.smithi097.stdout:266: done (15 left) 2024-04-18T21:52:49.661 INFO:tasks.rados.rados.0.smithi097.stdout:271: done (14 left) 2024-04-18T21:52:49.661 INFO:tasks.rados.rados.0.smithi097.stdout:273: done (13 left) 2024-04-18T21:52:49.661 INFO:tasks.rados.rados.0.smithi097.stdout:274: done (12 left) 2024-04-18T21:52:49.661 INFO:tasks.rados.rados.0.smithi097.stdout:276: done (11 left) 2024-04-18T21:52:49.661 INFO:tasks.rados.rados.0.smithi097.stdout:278: done (10 left) 2024-04-18T21:52:49.661 INFO:tasks.rados.rados.0.smithi097.stdout:280: done (9 left) 2024-04-18T21:52:49.662 INFO:tasks.rados.rados.0.smithi097.stdout:282: done (8 left) 2024-04-18T21:52:49.662 INFO:tasks.rados.rados.0.smithi097.stdout:284: done (7 left) 2024-04-18T21:52:49.662 INFO:tasks.rados.rados.0.smithi097.stdout:286: done (6 left) 2024-04-18T21:52:49.662 INFO:tasks.rados.rados.0.smithi097.stdout:287: done (5 left) 2024-04-18T21:52:49.662 INFO:tasks.rados.rados.0.smithi097.stdout:288: done (4 left) 2024-04-18T21:52:49.662 INFO:tasks.rados.rados.0.smithi097.stdout:290: done (3 left) 2024-04-18T21:52:49.662 INFO:tasks.rados.rados.0.smithi097.stdout:292: done (2 left) 2024-04-18T21:52:49.662 INFO:tasks.rados.rados.0.smithi097.stdout:295: snap_remove snap 14 2024-04-18T21:52:49.679 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:49.674+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:49.755 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:49.755 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:49.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.755 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.10s7 93/90 ECSubRead(tid=128, to_read={3:08dd949e:::smithi09731216-47:14=0,262144,0}, subchunks={3:08dd949e:::smithi09731216-47:14=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:49.755 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:49.683440+0000", 2024-04-18T21:52:49.755 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.072029592000000003, 2024-04-18T21:52:49.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.072079254999999995, 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.683440+0000", 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.683440+0000", 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:49.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.683441+0000", 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1829999999999999e-06 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.683463+0000", 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1827e-05 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.683491+0000", 2024-04-18T21:52:49.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7826000000000001e-05 2024-04-18T21:52:49.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.758 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:49.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:49.758 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:52:49.758 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 1 2024-04-18T21:52:49.758 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:49.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:49.813 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:49.810+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:49.881 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:49.879+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:49.937 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:49.937 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:49.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:49.982 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:49.978+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:50.080 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 77.341600305, 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61088117099999995, 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 35 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735191+0000", 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3699999999999999e-07 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735266+0000", 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4856000000000003e-05 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735279+0000", 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.36e-05 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735288+0000", 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0860000000000008e-06 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242298+0000", 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50700964999999998 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242568+0000", 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027023100000000001 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242974+0000", 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040533399999999998 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246576+0000", 2024-04-18T21:52:50.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036026919999999998 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345792+0000", 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099215503999999996 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345953+0000", 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161189 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346071+0000", 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118192 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 77.341487811999997, 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61108955600000003, 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 36 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.23e-07 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735310+0000", 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7560000000000002e-06 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735323+0000", 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735337+0000", 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4049999999999999e-05 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244243+0000", 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50890638099999996 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244422+0000", 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000178495 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244638+0000", 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216562 2024-04-18T21:52:50.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.247950+0000", 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033121750000000001 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346232+0000", 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098281564000000002 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346324+0000", 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2042999999999999e-05 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346392+0000", 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8052999999999994e-05 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:50.093 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 77.341438792999995, 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71224671699999997, 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 37 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0999999999999998e-07 2024-04-18T21:52:50.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735358+0000", 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2639999999999999e-06 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735367+0000", 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735375+0000", 2024-04-18T21:52:50.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5360000000000004e-06 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245345+0000", 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50996983699999998 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245615+0000", 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027018099999999998 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246022+0000", 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000407394 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.248536+0000", 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025132330000000001 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447444+0000", 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198908419 2024-04-18T21:52:50.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447532+0000", 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8010000000000006e-05 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447598+0000", 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6383000000000005e-05 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 72.428038279999996, 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61732750999999997, 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 117 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648753+0000", 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:52:50.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648758+0000", 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648768+0000", 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.414e-06 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648777+0000", 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.363e-06 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863479+0000", 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214701524 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863596+0000", 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117039 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863810+0000", 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021416800000000001 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.864846+0000", 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001036319 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265892+0000", 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40104641899999999 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265995+0000", 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010209600000000001 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.266080+0000", 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5049000000000005e-05 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.770864019000001, 2024-04-18T21:52:50.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60955713600000005, 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 161 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:50.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305927+0000", 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306056+0000", 2024-04-18T21:52:50.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012895300000000001 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306068+0000", 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306077+0000", 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9430000000000006e-06 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412179+0000", 2024-04-18T21:52:50.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106101709 2024-04-18T21:52:50.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412267+0000", 2024-04-18T21:52:50.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8251999999999995e-05 2024-04-18T21:52:50.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412388+0000", 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121136 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420472+0000", 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080833510000000008 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915141+0000", 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49466944899999998 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915356+0000", 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215305 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915484+0000", 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012713500000000001 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.770700196000007, 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61031149600000001, 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 162 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.112 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306091+0000", 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306097+0000", 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6300000000000003e-06 2024-04-18T21:52:50.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306106+0000", 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2199999999999998e-06 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306116+0000", 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8870000000000004e-06 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413869+0000", 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107753534 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414135+0000", 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026615200000000002 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414518+0000", 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038217299999999997 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420492+0000", 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059740879999999998 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915940+0000", 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49544827600000002 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916143+0000", 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000202752 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916402+0000", 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025901100000000002 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.770663826000003, 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61064147300000005, 2024-04-18T21:52:50.118 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 163 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.06e-07 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306136+0000", 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6999999999999997e-06 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306146+0000", 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8439999999999993e-06 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306154+0000", 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415593+0000", 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109439392 2024-04-18T21:52:50.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415833+0000", 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000240101 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416223+0000", 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000389804 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420508+0000", 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042851699999999996 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916537+0000", 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49602917499999999 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916669+0000", 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179899999999999 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916768+0000", 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8879e-05 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.770625292999995, 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61096294100000004, 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.124 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 164 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:50.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306166+0000", 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306275+0000", 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109749 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306287+0000", 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1182999999999999e-05 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306298+0000", 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1487e-05 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417044+0000", 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110745818 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417282+0000", 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000237712 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417665+0000", 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000383194 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420523+0000", 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028578010000000001 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916880+0000", 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49635731900000002 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917054+0000", 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017436200000000001 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917128+0000", 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3693000000000001e-05 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.129 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.770479297999998, 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70940268200000001, 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:50.130 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 165 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306312+0000", 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306317+0000", 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0699999999999997e-06 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306327+0000", 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7030000000000001e-06 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306349+0000", 2024-04-18T21:52:50.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2160000000000001e-05 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419233+0000", 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112884159 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419499+0000", 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265865 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419851+0000", 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035253999999999998 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.421040+0000", 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011886380000000001 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:50.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015628+0000", 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59458825299999996 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015677+0000", 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8470000000000002e-05 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015714+0000", 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7123999999999999e-05 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.135 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:438 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~622592] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.594715794000003, 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60190392599999998, 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 438 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482077+0000", 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3429999999999998e-06 2024-04-18T21:52:50.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482901+0000", 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082364099999999996 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482933+0000", 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2128999999999997e-05 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482949+0000", 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5872999999999999e-05 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676460+0000", 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19351131799999999 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676647+0000", 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018723700000000001 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.678595+0000", 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00194812 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.782086+0000", 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10349056299999999 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083745+0000", 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30165924199999999 2024-04-18T21:52:50.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083979+0000", 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023346 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:439 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~147456] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.593098308999998, 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70156516499999999, 2024-04-18T21:52:50.141 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 439 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483694+0000", 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2129999999999998e-06 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483902+0000", 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020762600000000001 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483934+0000", 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2140000000000001e-05 2024-04-18T21:52:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483950+0000", 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5792999999999999e-05 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779688+0000", 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29573833799999999 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779887+0000", 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019900000000000001 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.781116+0000", 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001229071 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.882567+0000", 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101451022 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185116+0000", 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.302548704 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185257+0000", 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014125800000000001 2024-04-18T21:52:50.147 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:440 3.fs0 3:fcd1a062:::smithi09731216-16:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.592814433999997, 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.10743576, 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.148 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 440 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483977+0000", 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7000000000000003e-07 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483982+0000", 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4e-06 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483994+0000", 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.134e-05 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484003+0000", 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1169999999999993e-06 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186403+0000", 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70240069199999999 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186624+0000", 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000220523 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.187505+0000", 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088087199999999999 2024-04-18T21:52:50.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188877+0000", 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001371723 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591138+0000", 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40226145699999999 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591412+0000", 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027366599999999999 2024-04-18T21:52:50.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:441 3.fs0 3:fcd1a062:::smithi09731216-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.592775191000001, 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.413060682, 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 441 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484016+0000", 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8299999999999996e-07 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484019+0000", 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3069999999999998e-06 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484038+0000", 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978999999999999e-05 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484047+0000", 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5939999999999994e-06 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188125+0000", 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70407843599999997 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188355+0000", 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022941899999999999 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692687+0000", 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50433201000000005 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692919+0000", 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023258299999999999 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.897076+0000", 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20415657100000001 2024-04-18T21:52:50.158 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:494 3.13s0 3:c8a99f7f:::smithi09731216-31:6 [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 29.581058224, 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59079312500000003, 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 494 2024-04-18T21:52:50.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495734+0000", 2024-04-18T21:52:50.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4929999999999999e-06 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495743+0000", 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5629999999999998e-06 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495768+0000", 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5147000000000002e-05 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495784+0000", 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6019e-05 2024-04-18T21:52:50.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.596290+0000", 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100505504 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.596489+0000", 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019867499999999999 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886268+0000", 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.28977927399999998 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886434+0000", 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000166288 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.086525+0000", 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20009116199999999 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:563 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~491520] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.655393306000001, 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62112892600000003, 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 563 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421400+0000", 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8729999999999999e-06 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422217+0000", 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081729399999999998 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422245+0000", 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7497000000000001e-05 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422262+0000", 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7601999999999999e-05 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422362+0000", 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9270000000000003e-05 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422471+0000", 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109067 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.423731+0000", 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012607779999999999 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.425601+0000", 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018698549999999999 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042390+0000", 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61678826399999997 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042526+0000", 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013642600000000001 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:564 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~589824] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.644823293000002, 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61084178300000003, 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.169 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 564 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431969+0000", 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.64e-06 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432549+0000", 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00058043400000000001 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432569+0000", 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0203999999999998e-05 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432578+0000", 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4619999999999996e-06 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432649+0000", 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0669999999999999e-05 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432762+0000", 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113474 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.433682+0000", 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092051399999999999 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.434979+0000", 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012963530000000001 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042699+0000", 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60771967500000001 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042809+0000", 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110357 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:565 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~557056] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:50.174 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.634732370999998, 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60103031100000004, 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 565 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442060+0000", 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.843e-06 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442728+0000", 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00066799899999999996 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442745+0000", 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6932999999999999e-05 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442753+0000", 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8029999999999997e-06 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.537940+0000", 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.095187356000000001 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.538060+0000", 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119986 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.539336+0000", 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012755990000000001 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546080+0000", 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067440950000000003 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042980+0000", 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49690054099999997 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.043088+0000", 2024-04-18T21:52:50.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108156 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:566 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~16384] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.634023087999999, 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60493431399999997, 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.180 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 566 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442768+0000", 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5899999999999995e-07 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442790+0000", 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1687000000000001e-05 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442802+0000", 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2564e-05 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442808+0000", 2024-04-18T21:52:50.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2830000000000001e-06 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540321+0000", 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097512961999999995 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540613+0000", 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029227700000000002 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.542271+0000", 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016580010000000001 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546084+0000", 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038128620000000002 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.047609+0000", 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50152518300000004 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.047702+0000", 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2736000000000001e-05 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:567 3.13s0 3:cd8bc96a:::smithi09731216-48:head [setxattr _header (57) in=7b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.633973623999999, 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70082249600000002, 2024-04-18T21:52:50.186 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 567 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9399999999999995e-07 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442820+0000", 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4229999999999999e-06 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442826+0000", 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0440000000000001e-06 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442830+0000", 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0640000000000004e-06 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.543400+0000", 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100569913 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.543930+0000", 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052991099999999999 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.545388+0000", 2024-04-18T21:52:50.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014587059999999999 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.646614+0000", 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101225551 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.143425+0000", 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.496811477 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.143639+0000", 2024-04-18T21:52:50.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021391300000000001 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:568 3.13s0 3:cd8bc96a:::smithi09731216-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.633955379, 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90286167399999995, 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:50.192 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 568 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999998e-07 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442837+0000", 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8020000000000001e-06 2024-04-18T21:52:50.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442844+0000", 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5869999999999998e-06 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442848+0000", 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9929999999999997e-06 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.545932+0000", 2024-04-18T21:52:50.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103084052 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546068+0000", 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013653800000000001 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.144727+0000", 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59865855499999998 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:50.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.144992+0000", 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026491399999999999 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.345697+0000", 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200704883 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:50.197 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:50.199 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:50.199 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:50.194+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:50.300 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:50.294+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:50.334 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:50.335 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:50.335 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:52:50.335 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:50.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:50.401 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:50.398+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:50.454 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:50.454 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:52:50.455 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-18T21:52:50.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-18T21:52:50.471 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:50.471 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:50.471 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:50.471 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:50.472 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:50.486 INFO:tasks.rados.rados.0.smithi097.stdout:295: done (2 left) 2024-04-18T21:52:50.486 INFO:tasks.rados.rados.0.smithi097.stdout:296: rollback oid 49 current snap is 23 2024-04-18T21:52:50.487 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 49 to 21 2024-04-18T21:52:50.487 INFO:tasks.rados.rados.0.smithi097.stdout:297: delete oid 34 current snap is 23 2024-04-18T21:52:50.488 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:50.502 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:50.498+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:50.603 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:50.599+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:50.615 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:50.615 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:50.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:50.615 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.046390273, 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201960743, 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:50.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566449+0000", 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.77e-06 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566495+0000", 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5488999999999998e-05 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566536+0000", 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0756000000000001e-05 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.666638+0000", 2024-04-18T21:52:50.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102466 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766859+0000", 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10022078500000001 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766875+0000", 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5806000000000001e-05 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768312+0000", 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014373389999999999 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768408+0000", 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6331999999999999e-05 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.620 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:118 3.17s0 3:eda737c1:::smithi09731216-40:head [append 0~671744,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 63.008147043999998, 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1801127490000001, 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 118 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604693+0000", 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2970000000000002e-06 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.605995+0000", 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001302276 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606021+0000", 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5320000000000002e-05 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606036+0000", 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5133000000000001e-05 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606232+0000", 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000196672 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606237+0000", 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7199999999999997e-06 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484281+0000", 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.87804420999999999 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484592+0000", 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031024299999999998 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.487837+0000", 2024-04-18T21:52:50.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003245571 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490891+0000", 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030539719999999998 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784613+0000", 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.293722237 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784804+0000", 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019009800000000001 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:119 3.17s0 3:eda737c1:::smithi09731216-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 63.006765195, 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.179191986, 2024-04-18T21:52:50.627 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 119 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606074+0000", 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.257e-06 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606080+0000", 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0100000000000001e-06 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606095+0000", 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5413e-05 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606104+0000", 2024-04-18T21:52:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0899999999999994e-06 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606286+0000", 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000181552 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606287+0000", 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.026e-06 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493720+0000", 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88743291700000004 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493872+0000", 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015190200000000001 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495103+0000", 2024-04-18T21:52:50.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00123149 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.496855+0000", 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017518189999999999 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785096+0000", 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.28824129900000001 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785265+0000", 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016821100000000001 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:120 3.17s0 3:eda737c1:::smithi09731216-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 63.006696065, 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4107226310000001, 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.634 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 120 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606143+0000", 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.16e-07 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606148+0000", 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4569999999999996e-06 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606163+0000", 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503e-05 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606175+0000", 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2036e-05 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706293+0000", 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100117875 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706297+0000", 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.126e-06 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495621+0000", 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78932349599999996 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495757+0000", 2024-04-18T21:52:50.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013642600000000001 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785995+0000", 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29023745499999998 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.786127+0000", 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013212200000000001 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.016864+0000", 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.23073769199999999 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.17s0 32/31 ECSubRead(tid=3, to_read={3:eda737c1:::smithi09731216-40:head=0,4096,0}, subchunks={3:eda737c1:::smithi09731216-40:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 61.825655064000003, 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.210779366, 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.640 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787250+0000", 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7152999999999999e-05 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887353+0000", 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102647 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997704+0000", 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110351217 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997962+0000", 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000258349 2024-04-18T21:52:50.643 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.17s0 32/31 ECSubReadReply(tid=3, attrs_read=0))", 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 61.816360060000001, 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200936119, 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796478+0000", 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796479+0000", 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8999999999999995e-07 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796491+0000", 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2065000000000001e-05 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796502+0000", 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.116e-05 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796509+0000", 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7170000000000008e-06 2024-04-18T21:52:50.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997338+0000", 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20082881799999999 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997414+0000", 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5568999999999994e-05 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/31 ECSubWrite(tid=28, reqid=client.4509.0:304, at_version=41'12, trim_to=0'0, roll_forward_to=32'8))", 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 52.443470497, 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201742379, 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169369+0000", 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6339999999999999e-06 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169413+0000", 2024-04-18T21:52:50.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3860999999999999e-05 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169487+0000", 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4578000000000003e-05 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.269583+0000", 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100095613 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369850+0000", 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10026700299999999 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369869+0000", 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9114e-05 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371022+0000", 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001152943 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371110+0000", 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7633000000000006e-05 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/26 ECSubWrite(tid=56, reqid=osd.10.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'25))", 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.827394472999998, 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201706881, 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.652 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785445+0000", 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7999999999999999e-06 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785499+0000", 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3730999999999998e-05 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785543+0000", 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3961000000000001e-05 2024-04-18T21:52:50.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785568+0000", 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.561e-05 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.985970+0000", 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200401511 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.986000+0000", 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0224000000000001e-05 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987085+0000", 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001085145 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987150+0000", 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4899000000000003e-05 2024-04-18T21:52:50.656 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 53/51 ECSubWrite(tid=2, reqid=client.4509.0:390, at_version=0'0, trim_to=0'0, roll_forward_to=53'7))", 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 40.914214610999998, 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201345094, 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698623+0000", 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698625+0000", 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2869999999999999e-06 2024-04-18T21:52:50.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698665+0000", 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9743000000000002e-05 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.698719+0000", 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4228999999999999e-05 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.798874+0000", 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100154506 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.799005+0000", 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013080200000000001 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.799030+0000", 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5058e-05 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.899930+0000", 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100900189 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.899968+0000", 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8279999999999999e-05 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.2s5 58/31 ECSubWrite(tid=87, reqid=client.4509.0:422, at_version=58'6, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:50.661 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 35.954651931999997, 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201964437, 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658187+0000", 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5069999999999999e-06 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658288+0000", 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010070500000000001 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658315+0000", 2024-04-18T21:52:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7348e-05 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658331+0000", 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5559000000000001e-05 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.758532+0000", 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100200734 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.758549+0000", 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6823000000000001e-05 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860062+0000", 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101513567 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860150+0000", 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8194000000000003e-05 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.2s5 58/31 ECSubWrite(tid=88, reqid=client.4509.0:423, at_version=58'7, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 35.954467674, 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20184664099999999, 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658371+0000", 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5400000000000001e-07 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658390+0000", 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8907e-05 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658404+0000", 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3974999999999999e-05 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658414+0000", 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0156000000000001e-05 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.759248+0000", 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10083389299999999 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.759263+0000", 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5101e-05 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860159+0000", 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10089624599999999 2024-04-18T21:52:50.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860217+0000", 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7809000000000002e-05 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.fs7 62/26 ECSubWrite(tid=203, reqid=osd.7.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=62'26))", 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:50.670 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.036631215, 2024-04-18T21:52:50.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20165132299999999, 2024-04-18T21:52:50.671 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.671 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.671 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.671 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576209+0000", 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9300000000000002e-06 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576234+0000", 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5287999999999999e-05 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576260+0000", 2024-04-18T21:52:50.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5808000000000001e-05 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.676352+0000", 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10009247 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.776560+0000", 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100207532 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.776579+0000", 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8936e-05 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.777791+0000", 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00121206 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.777858+0000", 2024-04-18T21:52:50.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7298999999999994e-05 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 64/26 ECSubWrite(tid=97, reqid=osd.5.0:97, at_version=64'55, trim_to=0'0, roll_forward_to=62'51))", 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.014653358, 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.202670817, 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.677 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598185+0000", 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598203+0000", 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8168e-05 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598214+0000", 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0509999999999999e-05 2024-04-18T21:52:50.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.698303+0000", 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100089224 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.798517+0000", 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100213922 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.798550+0000", 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3120999999999997e-05 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.800786+0000", 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002236334 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.800855+0000", 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8966000000000003e-05 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/54 ECSubWrite(tid=8, reqid=osd.9.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=64'13))", 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 29.729825951999999, 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30101452299999998, 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883014+0000", 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3800000000000001e-06 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883043+0000", 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8924999999999998e-05 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883085+0000", 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1674000000000001e-05 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883109+0000", 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4345000000000002e-05 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.083392+0000", 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200282457 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.083411+0000", 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9156000000000001e-05 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.183969+0000", 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100558567 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.184026+0000", 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7018999999999997e-05 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 64/26 ECSubWrite(tid=99, reqid=osd.5.0:97, at_version=0'0, trim_to=0'0, roll_forward_to=64'55))", 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 29.707080416, 2024-04-18T21:52:50.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201502976, 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905758+0000", 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.91e-07 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905764+0000", 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1399999999999997e-06 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905769+0000", 2024-04-18T21:52:50.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5029999999999997e-06 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905776+0000", 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6989999999999997e-06 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.106403+0000", 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20062649299999999 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.106420+0000", 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7187999999999999e-05 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.107159+0000", 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00073935799999999996 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.107260+0000", 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101304 2024-04-18T21:52:50.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es4 68/26 ECSubWrite(tid=58, reqid=osd.2.0:58, at_version=68'44, trim_to=0'0, roll_forward_to=66'40))", 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.230763063000001, 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.205606013, 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382077+0000", 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.863e-06 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382131+0000", 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.465e-05 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382164+0000", 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3081999999999999e-05 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.482269+0000", 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10010452 2024-04-18T21:52:50.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.582575+0000", 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100306485 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.582604+0000", 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8462e-05 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.587585+0000", 2024-04-18T21:52:50.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049807999999999996 2024-04-18T21:52:50.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.587681+0000", 2024-04-18T21:52:50.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6150999999999995e-05 2024-04-18T21:52:50.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.695 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s3 68/58 ECSubWrite(tid=215, reqid=osd.7.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.692568193, 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20176354299999999, 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920271+0000", 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8530000000000001e-06 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920293+0000", 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1926e-05 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920324+0000", 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1047999999999998e-05 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.020468+0000", 2024-04-18T21:52:50.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100143498 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.120740+0000", 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100271761 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.120767+0000", 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7268999999999999e-05 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.121955+0000", 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011884599999999999 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.122033+0000", 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7727999999999998e-05 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es4 73/72 ECSubWrite(tid=63, reqid=client.4509.0:562, at_version=73'48, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:50.700 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.022380+0000", 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 20.590458182999999, 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20326635300000001, 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022380+0000", 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022380+0000", 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022381+0000", 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022429+0000", 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8031000000000001e-05 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022462+0000", 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2555000000000003e-05 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022479+0000", 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6806000000000001e-05 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.222841+0000", 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200362389 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.222877+0000", 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6405000000000001e-05 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.225557+0000", 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026795249999999999 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.225646+0000", 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8988999999999998e-05 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.704 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s3 74/58 ECSubWrite(tid=222, reqid=client.4509.0:566, at_version=74'62, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.544653+0000", 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 20.068185028999999, 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.203329066, 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544653+0000", 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544653+0000", 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544654+0000", 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2299999999999999e-07 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544720+0000", 2024-04-18T21:52:50.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6229999999999994e-05 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544748+0000", 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7702e-05 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.644842+0000", 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100094432 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.745084+0000", 2024-04-18T21:52:50.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100242131 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.745111+0000", 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6418999999999999e-05 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.747885+0000", 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027739969999999998 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.747982+0000", 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7231999999999993e-05 2024-04-18T21:52:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "pg_update_log_missing(3.13s3 epoch 79/78 rep_tid 234 entries 79'66 (0'0) error 3:c8da3643:::smithi09731216-25:head by client.4509.0:610 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:35.805921+0000", 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 14.806917142, 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20149904199999999, 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.805921+0000", 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.805921+0000", 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.805922+0000", 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3829999999999999e-06 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.805956+0000", 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3958999999999998e-05 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.806003+0000", 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6752000000000002e-05 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.906116+0000", 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10011363400000001 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:36.006389+0000", 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10027270100000001 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:36.007420+0000", 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010306130000000001 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.712 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:50.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:50.714 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:50.710+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 94, 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "fsid": "9a7b119f-53a7-4e46-8d6e-229eac84ebee", 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "created": "2024-04-18T21:51:17.630312+0000", 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "modified": "2024-04-18T21:52:50.477668+0000", 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "last_up_change": "2024-04-18T21:51:29.769135+0000", 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "last_in_change": "2024-04-18T21:52:45.415971+0000", 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_num": 5799936, 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_set": [ 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", 2024-04-18T21:52:50.767 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "sortbitwise" 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_version": 6, 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "full_ratio": 0.94999998807907104, 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_snapshot": "", 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_max": 3, 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "max_osd": 12, 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "require_min_compat_client": "luminous", 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "min_compat_client": "jewel", 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "require_osd_release": "squid", 2024-04-18T21:52:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_crimson": false, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "pools": [ 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 1, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_name": ".mgr", 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "create_time": "2024-04-18T21:51:28.321879+0000", 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_names": "hashpspool", 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "type": 1, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 2, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "min_size": 1, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_rule": 0, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, 2024-04-18T21:52:50.769 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "object_hash": 2, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscale_mode": "off", 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num": 1, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num": 1, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num_target": 1, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_target": 1, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_pending": 1, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "last_pg_merge_meta": { 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "source_pgid": "0.0", 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "ready_epoch": 0, 2024-04-18T21:52:50.770 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_started": 0, 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_clean": 0, 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "source_version": "0'0", 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "target_version": "0'0" 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "last_change": "20", 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend": "0", 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_preluminous": "0", 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "auid": 0, 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_mode": "selfmanaged", 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_seq": 0, 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_epoch": 0, 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_snaps": [], 2024-04-18T21:52:50.771 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "tiers": [], 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "tier_of": -1, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "read_tier": -1, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "write_tier": -1, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_mode": "none", 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_bytes": 0, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_objects": 0, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_flush_age": 0, 2024-04-18T21:52:50.772 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_evict_age": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_profile": "", 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_params": { 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "none" 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_period": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_count": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "use_gmt_hitset": true, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "min_read_recency_for_promote": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "grade_table": [], 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "stripe_width": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "expected_num_objects": 0, 2024-04-18T21:52:50.773 INFO:teuthology.orchestra.run.smithi097.stdout: "fast_read": false, 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "options": { 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_max": 32, 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_min": 1 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "application_metadata": { 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": {} 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "read_balance": { 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "score_type": "Fair distribution", 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "score_acting": 11.760000228881836, 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "score_stable": 11.760000228881836, 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "optimal_score": 0.17000000178813934, 2024-04-18T21:52:50.774 INFO:teuthology.orchestra.run.smithi097.stdout: "raw_score_acting": 2, 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "raw_score_stable": 2, 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "average_primary_affinity": 1, 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "average_primary_affinity_weighted": 1 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 2, 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_name": "rbd", 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "create_time": "2024-04-18T21:51:30.835497+0000", 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-04-18T21:52:50.775 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_names": "hashpspool", 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "type": 1, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 2, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "min_size": 1, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_rule": 0, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "object_hash": 2, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscale_mode": "off", 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num": 8, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num": 8, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num_target": 8, 2024-04-18T21:52:50.776 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_target": 8, 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_pending": 8, 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "last_pg_merge_meta": { 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "source_pgid": "0.0", 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "ready_epoch": 0, 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_started": 0, 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_clean": 0, 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "source_version": "0'0", 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "target_version": "0'0" 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "last_change": "23", 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend": "0", 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_preluminous": "0", 2024-04-18T21:52:50.777 INFO:teuthology.orchestra.run.smithi097.stdout: "auid": 0, 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_mode": "selfmanaged", 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_seq": 0, 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_epoch": 0, 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_snaps": [], 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "tiers": [], 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "tier_of": -1, 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "read_tier": -1, 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "write_tier": -1, 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_mode": "none", 2024-04-18T21:52:50.778 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_bytes": 0, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_objects": 0, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_flush_age": 0, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_evict_age": 0, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_profile": "", 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_params": { 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "none" 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_period": 0, 2024-04-18T21:52:50.779 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_count": 0, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "use_gmt_hitset": true, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "min_read_recency_for_promote": 0, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "grade_table": [], 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "stripe_width": 0, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "expected_num_objects": 0, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "fast_read": false, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "options": {}, 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "application_metadata": { 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd": {} 2024-04-18T21:52:50.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "read_balance": { 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "score_type": "Fair distribution", 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "score_acting": 2.9900000095367432, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "score_stable": 2.9900000095367432, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "optimal_score": 0.67000001668930054, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "raw_score_acting": 2, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "raw_score_stable": 2, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "average_primary_affinity": 1, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: "average_primary_affinity_weighted": 1 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 3, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_name": "unique_pool_0", 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "create_time": "2024-04-18T21:51:40.934866+0000", 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 8193, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "type": 3, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 8, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "min_size": 5, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_rule": 1, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-18T21:52:50.782 INFO:teuthology.orchestra.run.smithi097.stdout: "object_hash": 2, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscale_mode": "off", 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num": 24, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num": 21, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_placement_num_target": 16, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_target": 16, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num_pending": 24, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "last_pg_merge_meta": { 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "source_pgid": "3.18", 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "ready_epoch": 86, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_started": 85, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "last_epoch_clean": 85, 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "source_version": "81'26", 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: "target_version": "30'2" 2024-04-18T21:52:50.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "last_change": "94", 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend": "0", 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_prenautilus": "92", 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "last_force_op_resend_preluminous": "88", 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "auid": 0, 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_mode": "selfmanaged", 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_seq": 24, 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_epoch": 94, 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_snaps": [], 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, 2024-04-18T21:52:50.784 INFO:teuthology.orchestra.run.smithi097.stdout: "tiers": [], 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "tier_of": -1, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "read_tier": -1, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "write_tier": -1, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_mode": "none", 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_bytes": 0, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "target_max_objects": 0, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_flush_age": 0, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "cache_min_evict_age": 0, 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_profile": "lrcprofile", 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_params": { 2024-04-18T21:52:50.785 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "none" 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_period": 0, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_count": 0, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "use_gmt_hitset": true, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "min_read_recency_for_promote": 0, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "grade_table": [], 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "stripe_width": 16384, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "expected_num_objects": 0, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "fast_read": false, 2024-04-18T21:52:50.786 INFO:teuthology.orchestra.run.smithi097.stdout: "options": {}, 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "application_metadata": { 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "rados": {} 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 0, 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "2a27387f-73d5-4bf2-bb86-80c069a1d6f8", 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 0, 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 0, 2024-04-18T21:52:50.787 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 42, 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6808", 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6812", 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6817", 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6820", 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6830", 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:52:50.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6831", 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6824", 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6828", 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3458443931 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.97:6812/3458443931", 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.97:6820/3458443931", 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.97:6831/3458443931", 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.97:6828/3458443931", 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 1, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "2f6c823f-4b24-4ecf-9561-710723acded2", 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 89, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.793 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6800", 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6801", 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6802", 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6803", 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:52:50.795 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6811", 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6815", 2024-04-18T21:52:50.796 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6805", 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6807", 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1179456160 2024-04-18T21:52:50.797 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.97:6801/1179456160", 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.97:6803/1179456160", 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.97:6815/1179456160", 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.97:6807/1179456160", 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.798 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 2, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "05866927-b7ea-44e6-8a5e-f23263011fa4", 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 90, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.799 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6809", 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6813", 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6816", 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6819", 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6825", 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6827", 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.802 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6821", 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6823", 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 598211345 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.97:6813/598211345", 2024-04-18T21:52:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.97:6819/598211345", 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.97:6827/598211345", 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.97:6823/598211345", 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 3, 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "1fbdd117-432a-4bef-879e-96823b61820a", 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.804 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 90, 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6804", 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.805 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6806", 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6810", 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6814", 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6826", 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6829", 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6818", 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:52:50.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6822", 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 40393780 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.97:6806/40393780", 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.97:6814/40393780", 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.97:6829/40393780", 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.97:6822/40393780", 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.809 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 4, 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "a7f8da99-45f2-4d8f-8c8c-efd9632b0f61", 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.810 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 93, 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6808", 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.811 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6809", 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6810", 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.812 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6811", 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6814", 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.813 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6815", 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6812", 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:52:50.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6813", 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3973913623 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.104:6809/3973913623", 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.104:6811/3973913623", 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.104:6815/3973913623", 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.104:6813/3973913623", 2024-04-18T21:52:50.815 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 5, 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "2230dfaa-7c85-469b-a95b-594960d77b28", 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.816 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 90, 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6800", 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6801", 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6802", 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:52:50.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6803", 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6806", 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:52:50.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6807", 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.820 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6804", 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6805", 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1478992259 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.104:6801/1478992259", 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.104:6803/1478992259", 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.104:6807/1478992259", 2024-04-18T21:52:50.821 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.104:6805/1478992259", 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 6, 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "e7437af9-b161-482e-8939-4d2b4c2da10e", 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.822 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 93, 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6816", 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6817", 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6818", 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:52:50.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6819", 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6822", 2024-04-18T21:52:50.825 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6823", 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.826 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6820", 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6821", 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 462814955 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.104:6817/462814955", 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.104:6819/462814955", 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.104:6823/462814955", 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.104:6821/462814955", 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.827 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 7, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "a062c3b3-7e00-4bdf-9731-0072826522c9", 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 0, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 0, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:52:50.828 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 87, 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6824", 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6825", 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6826", 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6827", 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.830 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6830", 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6831", 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.831 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6828", 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.104:6829", 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 581913590 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.832 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.104:6825/581913590", 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.104:6827/581913590", 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.104:6831/581913590", 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.104:6829/581913590", 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 8, 2024-04-18T21:52:50.833 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "f2473c34-2db3-4806-a6ea-bb46f42c4ab8", 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 17, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 92, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6804", 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6806", 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.835 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6808", 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6812", 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.836 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6822", 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6825", 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.837 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6816", 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6819", 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1020504720 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.183:6806/1020504720", 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.183:6812/1020504720", 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.183:6825/1020504720", 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.183:6819/1020504720", 2024-04-18T21:52:50.838 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 9, 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "f08d755f-cc20-468c-b3ca-cfb88b44bdf1", 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.839 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 18, 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 90, 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6810", 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6814", 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6818", 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6821", 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6829", 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:52:50.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6830", 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6824", 2024-04-18T21:52:50.843 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6827", 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1194329140 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.183:6814/1194329140", 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.183:6821/1194329140", 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.183:6830/1194329140", 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.183:6827/1194329140", 2024-04-18T21:52:50.844 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 10, 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "247fad9b-8847-416f-bfbb-9d8b466c3aa6", 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.845 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 19, 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 90, 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6809", 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6813", 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6817", 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.847 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6820", 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6828", 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.848 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6831", 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6823", 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.849 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6826", 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1192340806 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.183:6813/1192340806", 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.183:6820/1192340806", 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.183:6831/1192340806", 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.183:6826/1192340806", 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.850 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 11, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "uuid": "4a51c1be-4125-451a-a910-c8e364fe21be", 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "up": 1, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "in": 1, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "weight": 1, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_affinity": 1, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 0, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 0, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 17, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "up_thru": 90, 2024-04-18T21:52:50.851 INFO:teuthology.orchestra.run.smithi097.stdout: "down_at": 0, 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "lost_at": 0, 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addrs": { 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6800", 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6801", 2024-04-18T21:52:50.852 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6802", 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.853 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6803", 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addrs": { 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6811", 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.854 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6815", 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addrs": { 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6805", 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.855 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.183:6807", 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3144695911 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "public_addr": "172.21.15.183:6801/3144695911", 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.183:6803/3144695911", 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.183:6815/3144695911", 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.183:6807/3144695911", 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "state": [ 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "exists", 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: "up" 2024-04-18T21:52:50.856 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_xinfo": [ 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 0, 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 65536, 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:32.843393+0000", 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 1, 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.965087+0000", 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 2, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:51:26.895463+0000", 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 3, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:45.523740+0000", 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 4, 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:35.646618+0000", 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 5, 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:43.513068+0000", 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 6, 2024-04-18T21:52:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:36.263252+0000", 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 7, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 65536, 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:28.081472+0000", 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 8, 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:42.792903+0000", 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 9, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:38.930203+0000", 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 10, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:35.599364+0000", 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": 11, 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "down_stamp": "0.000000", 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_probability": 0, 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "laggy_interval": 0, 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "features": 4540701547738038271, 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "old_weight": 0, 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "last_purged_snaps_scrub": "2024-04-18T21:52:47.013093+0000", 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_upmap": [], 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_upmap_items": [], 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_upmap_primaries": [], 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_temp": [ 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.4", 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: 2, 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: 9, 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 6 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.5", 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 6, 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 11, 2024-04-18T21:52:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 9 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.6", 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 11, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: 5 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.9", 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: 2, 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: 7, 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: 9, 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: 6 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.b", 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 6 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.e", 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 2, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 6, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 11, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 9, 2024-04-18T21:52:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.f", 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: 2, 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: 11, 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: 8 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.10", 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: 6, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: 2, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: 11, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.11", 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: 6, 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: 2, 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.12", 2024-04-18T21:52:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 9, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 6, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 1 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.13", 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 6, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 11, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 9 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.15", 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 5, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: 6, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 11, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 9 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.16", 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 11, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 5 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "pgid": "3.17", 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "osds": [ 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 8, 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 3, 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 2147483647, 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 10, 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 9, 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 1, 2024-04-18T21:52:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: 4, 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: 5 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "primary_temp": [], 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "blocklist": {}, 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "range_blocklist": {}, 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "erasure_code_profiles": { 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "default": { 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-failure-domain": "osd", 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "k": "2", 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "m": "1", 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "plugin": "jerasure", 2024-04-18T21:52:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "technique": "reed_sol_van" 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "lrcprofile": { 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-device-class": "", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-failure-domain": "osd", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-num-failure-domains": "0", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-osds-per-failure-domain": "0", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-root": "default", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "k": "4", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "l": "3", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "m": "2", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "lrcprofile", 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "plugin": "lrc" 2024-04-18T21:52:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps_queue": [ 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 3, 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "snaps": [ 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "begin": 6, 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "length": 1 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "begin": 10, 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "length": 1 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "begin": 16, 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "length": 3 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "begin": 20, 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "length": 3 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "begin": 24, 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "length": 1 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "new_removed_snaps": [ 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "pool": 3, 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "snaps": [ 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "begin": 16, 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "length": 1 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "new_purged_snaps": [], 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_node_flags": {}, 2024-04-18T21:52:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "device_class_flags": {}, 2024-04-18T21:52:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "stretch_mode": { 2024-04-18T21:52:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "stretch_mode_enabled": false, 2024-04-18T21:52:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "stretch_bucket_count": 0, 2024-04-18T21:52:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "degraded_stretch_mode": 0, 2024-04-18T21:52:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "recovering_stretch_mode": 0, 2024-04-18T21:52:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "stretch_mode_bucket": 0 2024-04-18T21:52:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:52:50.884 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:50.886 INFO:tasks.thrashosds.thrasher:j is {'epoch': 94, 'fsid': '9a7b119f-53a7-4e46-8d6e-229eac84ebee', 'created': '2024-04-18T21:51:17.630312+0000', 'modified': '2024-04-18T21:52:50.477668+0000', 'last_up_change': '2024-04-18T21:51:29.769135+0000', 'last_in_change': '2024-04-18T21:52:45.415971+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T21:51:28.321879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-18T21:51:30.835497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-18T21:51:40.934866+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 8, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 24, 'pg_placement_num': 21, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 24, 'last_pg_merge_meta': {'source_pgid': '3.18', 'ready_epoch': 86, 'last_epoch_started': 85, 'last_epoch_clean': 85, 'source_version': "81'26", 'target_version': "30'2"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '92', 'last_force_op_resend_preluminous': '88', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 24, 'snap_epoch': 94, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'lrcprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '2a27387f-73d5-4bf2-bb86-80c069a1d6f8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 42, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 3458443931}, {'type': 'v1', 'addr': '172.21.15.97:6812', 'nonce': 3458443931}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6817', 'nonce': 3458443931}, {'type': 'v1', 'addr': '172.21.15.97:6820', 'nonce': 3458443931}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6830', 'nonce': 3458443931}, {'type': 'v1', 'addr': '172.21.15.97:6831', 'nonce': 3458443931}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6824', 'nonce': 3458443931}, {'type': 'v1', 'addr': '172.21.15.97:6828', 'nonce': 3458443931}]}, 'public_addr': '172.21.15.97:6812/3458443931', 'cluster_addr': '172.21.15.97:6820/3458443931', 'heartbeat_back_addr': '172.21.15.97:6831/3458443931', 'heartbeat_front_addr': '172.21.15.97:6828/3458443931', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f6c823f-4b24-4ecf-9561-710723acded2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 1179456160}, {'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 1179456160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 1179456160}, {'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 1179456160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 1179456160}, {'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 1179456160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 1179456160}, {'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 1179456160}]}, 'public_addr': '172.21.15.97:6801/1179456160', 'cluster_addr': '172.21.15.97:6803/1179456160', 'heartbeat_back_addr': '172.21.15.97:6815/1179456160', 'heartbeat_front_addr': '172.21.15.97:6807/1179456160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '05866927-b7ea-44e6-8a5e-f23263011fa4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 598211345}, {'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 598211345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6816', 'nonce': 598211345}, {'type': 'v1', 'addr': '172.21.15.97:6819', 'nonce': 598211345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6825', 'nonce': 598211345}, {'type': 'v1', 'addr': '172.21.15.97:6827', 'nonce': 598211345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6821', 'nonce': 598211345}, {'type': 'v1', 'addr': '172.21.15.97:6823', 'nonce': 598211345}]}, 'public_addr': '172.21.15.97:6813/598211345', 'cluster_addr': '172.21.15.97:6819/598211345', 'heartbeat_back_addr': '172.21.15.97:6827/598211345', 'heartbeat_front_addr': '172.21.15.97:6823/598211345', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1fbdd117-432a-4bef-879e-96823b61820a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 40393780}, {'type': 'v1', 'addr': '172.21.15.97:6806', 'nonce': 40393780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 40393780}, {'type': 'v1', 'addr': '172.21.15.97:6814', 'nonce': 40393780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6826', 'nonce': 40393780}, {'type': 'v1', 'addr': '172.21.15.97:6829', 'nonce': 40393780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6818', 'nonce': 40393780}, {'type': 'v1', 'addr': '172.21.15.97:6822', 'nonce': 40393780}]}, 'public_addr': '172.21.15.97:6806/40393780', 'cluster_addr': '172.21.15.97:6814/40393780', 'heartbeat_back_addr': '172.21.15.97:6829/40393780', 'heartbeat_front_addr': '172.21.15.97:6822/40393780', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a7f8da99-45f2-4d8f-8c8c-efd9632b0f61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 93, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6808', 'nonce': 3973913623}, {'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 3973913623}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6810', 'nonce': 3973913623}, {'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 3973913623}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6814', 'nonce': 3973913623}, {'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 3973913623}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6812', 'nonce': 3973913623}, {'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 3973913623}]}, 'public_addr': '172.21.15.104:6809/3973913623', 'cluster_addr': '172.21.15.104:6811/3973913623', 'heartbeat_back_addr': '172.21.15.104:6815/3973913623', 'heartbeat_front_addr': '172.21.15.104:6813/3973913623', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2230dfaa-7c85-469b-a95b-594960d77b28', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6800', 'nonce': 1478992259}, {'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1478992259}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6802', 'nonce': 1478992259}, {'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1478992259}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6806', 'nonce': 1478992259}, {'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 1478992259}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6804', 'nonce': 1478992259}, {'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 1478992259}]}, 'public_addr': '172.21.15.104:6801/1478992259', 'cluster_addr': '172.21.15.104:6803/1478992259', 'heartbeat_back_addr': '172.21.15.104:6807/1478992259', 'heartbeat_front_addr': '172.21.15.104:6805/1478992259', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7437af9-b161-482e-8939-4d2b4c2da10e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 93, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6816', 'nonce': 462814955}, {'type': 'v1', 'addr': '172.21.15.104:6817', 'nonce': 462814955}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6818', 'nonce': 462814955}, {'type': 'v1', 'addr': '172.21.15.104:6819', 'nonce': 462814955}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6822', 'nonce': 462814955}, {'type': 'v1', 'addr': '172.21.15.104:6823', 'nonce': 462814955}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6820', 'nonce': 462814955}, {'type': 'v1', 'addr': '172.21.15.104:6821', 'nonce': 462814955}]}, 'public_addr': '172.21.15.104:6817/462814955', 'cluster_addr': '172.21.15.104:6819/462814955', 'heartbeat_back_addr': '172.21.15.104:6823/462814955', 'heartbeat_front_addr': '172.21.15.104:6821/462814955', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a062c3b3-7e00-4bdf-9731-0072826522c9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 87, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6824', 'nonce': 581913590}, {'type': 'v1', 'addr': '172.21.15.104:6825', 'nonce': 581913590}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6826', 'nonce': 581913590}, {'type': 'v1', 'addr': '172.21.15.104:6827', 'nonce': 581913590}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6830', 'nonce': 581913590}, {'type': 'v1', 'addr': '172.21.15.104:6831', 'nonce': 581913590}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6828', 'nonce': 581913590}, {'type': 'v1', 'addr': '172.21.15.104:6829', 'nonce': 581913590}]}, 'public_addr': '172.21.15.104:6825/581913590', 'cluster_addr': '172.21.15.104:6827/581913590', 'heartbeat_back_addr': '172.21.15.104:6831/581913590', 'heartbeat_front_addr': '172.21.15.104:6829/581913590', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f2473c34-2db3-4806-a6ea-bb46f42c4ab8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 1020504720}, {'type': 'v1', 'addr': '172.21.15.183:6806', 'nonce': 1020504720}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 1020504720}, {'type': 'v1', 'addr': '172.21.15.183:6812', 'nonce': 1020504720}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 1020504720}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 1020504720}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 1020504720}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 1020504720}]}, 'public_addr': '172.21.15.183:6806/1020504720', 'cluster_addr': '172.21.15.183:6812/1020504720', 'heartbeat_back_addr': '172.21.15.183:6825/1020504720', 'heartbeat_front_addr': '172.21.15.183:6819/1020504720', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'f08d755f-cc20-468c-b3ca-cfb88b44bdf1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 1194329140}, {'type': 'v1', 'addr': '172.21.15.183:6814', 'nonce': 1194329140}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 1194329140}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 1194329140}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6829', 'nonce': 1194329140}, {'type': 'v1', 'addr': '172.21.15.183:6830', 'nonce': 1194329140}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 1194329140}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 1194329140}]}, 'public_addr': '172.21.15.183:6814/1194329140', 'cluster_addr': '172.21.15.183:6821/1194329140', 'heartbeat_back_addr': '172.21.15.183:6830/1194329140', 'heartbeat_front_addr': '172.21.15.183:6827/1194329140', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '247fad9b-8847-416f-bfbb-9d8b466c3aa6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6809', 'nonce': 1192340806}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 1192340806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6817', 'nonce': 1192340806}, {'type': 'v1', 'addr': '172.21.15.183:6820', 'nonce': 1192340806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 1192340806}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 1192340806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6823', 'nonce': 1192340806}, {'type': 'v1', 'addr': '172.21.15.183:6826', 'nonce': 1192340806}]}, 'public_addr': '172.21.15.183:6813/1192340806', 'cluster_addr': '172.21.15.183:6820/1192340806', 'heartbeat_back_addr': '172.21.15.183:6831/1192340806', 'heartbeat_front_addr': '172.21.15.183:6826/1192340806', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '4a51c1be-4125-451a-a910-c8e364fe21be', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6800', 'nonce': 3144695911}, {'type': 'v1', 'addr': '172.21.15.183:6801', 'nonce': 3144695911}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 3144695911}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 3144695911}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6811', 'nonce': 3144695911}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3144695911}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6805', 'nonce': 3144695911}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 3144695911}]}, 'public_addr': '172.21.15.183:6801/3144695911', 'cluster_addr': '172.21.15.183:6803/3144695911', 'heartbeat_back_addr': '172.21.15.183:6815/3144695911', 'heartbeat_front_addr': '172.21.15.183:6807/3144695911', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-18T21:52:32.843393+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.965087+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:51:26.895463+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:52:45.523740+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:52:35.646618+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:52:43.513068+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:52:36.263252+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-18T21:52:28.081472+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:52:42.792903+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:52:38.930203+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:52:35.599364+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-18T21:52:47.013093+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.4', 'osds': [10, 2, 1, 5, 8, 2147483647, 9, 6]}, {'pgid': '3.5', 'osds': [5, 8, 10, 6, 2147483647, 11, 3, 9]}, {'pgid': '3.6', 'osds': [11, 3, 4, 2147483647, 10, 2147483647, 1, 5]}, {'pgid': '3.9', 'osds': [3, 2, 7, 1, 5, 9, 4, 6]}, {'pgid': '3.b', 'osds': [10, 3, 4, 2147483647, 1, 2147483647, 8, 6]}, {'pgid': '3.e', 'osds': [2, 6, 1, 4, 8, 11, 9, 2147483647]}, {'pgid': '3.f', 'osds': [3, 1, 10, 2147483647, 2, 5, 11, 8]}, {'pgid': '3.10', 'osds': [6, 2, 5, 11, 8, 4, 10, 2147483647]}, {'pgid': '3.11', 'osds': [3, 4, 6, 10, 1, 5, 2, 2147483647]}, {'pgid': '3.12', 'osds': [9, 2147483647, 8, 10, 4, 5, 6, 1]}, {'pgid': '3.13', 'osds': [2147483647, 5, 4, 8, 6, 11, 1, 9]}, {'pgid': '3.15', 'osds': [5, 8, 10, 6, 2147483647, 11, 3, 9]}, {'pgid': '3.16', 'osds': [11, 3, 4, 2147483647, 10, 2147483647, 1, 5]}, {'pgid': '3.17', 'osds': [8, 3, 2147483647, 10, 9, 1, 4, 5]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'lrcprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'l': '3', 'm': '2', 'name': 'lrcprofile', 'plugin': 'lrc'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 6, 'length': 1}, {'begin': 10, 'length': 1}, {'begin': 16, 'length': 3}, {'begin': 20, 'length': 3}, {'begin': 24, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 16, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-18T21:52:50.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:52:50.887 INFO:tasks.rados.rados.0.smithi097.stdout:296: finishing rollback tid 0 to smithi09731216-49 2024-04-18T21:52:50.887 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:50.882+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:50.889 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.5s7 94/90 ECSubRead(tid=187, to_read={3:a2c4e662:::smithi09731216-50:18=0,262144,0}, subchunks={3:a2c4e662:::smithi09731216-50:18=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.799969+0000", 2024-04-18T21:52:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.089803464, 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.089854869000000004, 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.799969+0000", 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.799969+0000", 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.799970+0000", 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0999999999999997e-07 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.799984+0000", 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3337e-05 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.800004+0000", 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0424e-05 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:732 3.12s0 3:4c1ede14:::smithi09731216-49:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.40178444499999999, 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40192423799999999, 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 732 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487989+0000", 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8299999999999997e-07 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487992+0000", 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.253e-06 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488000+0000", 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7090000000000002e-06 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488006+0000", 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3550000000000001e-06 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488154+0000", 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014786700000000001 2024-04-18T21:52:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488167+0000", 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2836e-05 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.499213+0000", 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011045981999999999 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.499412+0000", 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000198519 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.500214+0000", 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00080208099999999995 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.503072+0000", 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00285819 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=22, last_complete=94'21, committed=1, applied=1))", 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.702823+0000", 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.186950319, 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18726910599999999, 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702823+0000", 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702823+0000", 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702823+0000", 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.53e-07 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702826+0000", 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.18e-06 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702835+0000", 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8000000000000004e-06 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702844+0000", 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2089999999999994e-06 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=22, last_complete=64'13, committed=1, applied=1))", 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.702894+0000", 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.18687937399999999, 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18726985199999999, 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702894+0000", 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702894+0000", 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702894+0000", 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.53e-07 2024-04-18T21:52:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702897+0000", 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8270000000000002e-06 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702912+0000", 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5172e-05 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702946+0000", 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3958999999999998e-05 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 94/88 ECSubWrite(tid=83, reqid=osd.1.0:82, at_version=0'0, trim_to=0'0, roll_forward_to=94'46))", 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.710166+0000", 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.17960733000000001, 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.18007429999999999, 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710166+0000", 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710166+0000", 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710167+0000", 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7570000000000001e-06 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710187+0000", 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0102000000000002e-05 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710214+0000", 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7056e-05 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710238+0000", 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3093000000000001e-05 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.810421+0000", 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10018368 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.810430+0000", 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5210000000000002e-06 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.8s4 94/86 ECSubWrite(tid=145, reqid=osd.3.0:144, at_version=0'0, trim_to=0'0, roll_forward_to=94'30))", 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.725305+0000", 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.16446778000000001, 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16502383700000001, 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.725305+0000", 2024-04-18T21:52:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.725305+0000", 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.725306+0000", 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.266e-06 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.725325+0000", 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8984e-05 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.725348+0000", 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2699000000000001e-05 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.725363+0000", 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5102999999999999e-05 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 6 2024-04-18T21:52:50.911 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:50.912 INFO:tasks.rados.rados.0.smithi097.stdout:297: done (3 left) 2024-04-18T21:52:50.912 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 35 current snap is 23 2024-04-18T21:52:50.912 INFO:tasks.rados.rados.0.smithi097.stdout:298: seq_num 118 ranges {1753088=442368,2195456=753664,2949120=655360,3604480=147456} 2024-04-18T21:52:50.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:50.933 INFO:tasks.rados.rados.0.smithi097.stdout:298: writing smithi09731216-35 from 1753088 to 2195456 tid 1 2024-04-18T21:52:50.946 INFO:tasks.rados.rados.0.smithi097.stdout:298: writing smithi09731216-35 from 2195456 to 2949120 tid 2 2024-04-18T21:52:50.955 INFO:tasks.rados.rados.0.smithi097.stdout:298: writing smithi09731216-35 from 2949120 to 3604480 tid 3 2024-04-18T21:52:50.957 INFO:tasks.rados.rados.0.smithi097.stdout:298: writing smithi09731216-35 from 3604480 to 3751936 tid 4 2024-04-18T21:52:50.957 INFO:tasks.rados.rados.0.smithi097.stdout:299: rmattr oid 9 current snap is 23 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:299: done (4 left) 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:300: read oid 46 snap 21 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:300: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:301: read oid 50 snap -1 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:301: expect (ObjNum 114 snap 23 seq_num 114) 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 13 current snap is 23 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:296: finishing rollback tid 1 to smithi09731216-49 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 49 v 21 (ObjNum 129 snap 0 seq_num 1748268736) dirty dne 2024-04-18T21:52:50.958 INFO:tasks.rados.rados.0.smithi097.stdout:302: seq_num 119 ranges {0=458752,458752=507904,966656=507904,1474560=458752,1933312=81920} 2024-04-18T21:52:50.962 INFO:tasks.rados.rados.0.smithi097.stdout:302: writing smithi09731216-13 from 0 to 458752 tid 1 2024-04-18T21:52:50.968 INFO:tasks.rados.rados.0.smithi097.stdout:302: writing smithi09731216-13 from 458752 to 966656 tid 2 2024-04-18T21:52:50.974 INFO:tasks.rados.rados.0.smithi097.stdout:302: writing smithi09731216-13 from 966656 to 1474560 tid 3 2024-04-18T21:52:50.980 INFO:tasks.rados.rados.0.smithi097.stdout:302: writing smithi09731216-13 from 1474560 to 1933312 tid 4 2024-04-18T21:52:50.981 INFO:tasks.rados.rados.0.smithi097.stdout:302: writing smithi09731216-13 from 1933312 to 2015232 tid 5 2024-04-18T21:52:50.981 INFO:tasks.rados.rados.0.smithi097.stdout:296: done (6 left) 2024-04-18T21:52:50.981 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 11 current snap is 23 2024-04-18T21:52:50.981 INFO:tasks.rados.rados.0.smithi097.stdout:303: seq_num 120 ranges {6160384=475136,6635520=655360,7290880=622592,7913472=278528} 2024-04-18T21:52:50.986 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:50.983+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:51.032 INFO:tasks.rados.rados.0.smithi097.stdout:303: writing smithi09731216-11 from 6160384 to 6635520 tid 1 2024-04-18T21:52:51.037 INFO:tasks.rados.rados.0.smithi097.stdout:303: writing smithi09731216-11 from 6635520 to 7290880 tid 2 2024-04-18T21:52:51.037 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:51.037 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:51.037 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:51.037 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:51.038 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:51.040 INFO:tasks.rados.rados.0.smithi097.stdout:303: writing smithi09731216-11 from 7290880 to 7913472 tid 3 2024-04-18T21:52:51.043 INFO:tasks.rados.rados.0.smithi097.stdout:303: writing smithi09731216-11 from 7913472 to 8192000 tid 4 2024-04-18T21:52:51.043 INFO:tasks.rados.rados.0.smithi097.stdout:304: read oid 45 snap 21 2024-04-18T21:52:51.043 INFO:tasks.rados.rados.0.smithi097.stdout:304: expect (ObjNum 84 snap 14 seq_num 84) 2024-04-18T21:52:51.043 INFO:tasks.rados.rados.0.smithi097.stdout:305: rmattr oid 16 current snap is 23 2024-04-18T21:52:51.043 INFO:tasks.rados.rados.0.smithi097.stdout:305: done (8 left) 2024-04-18T21:52:51.043 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 49 current snap is 23 2024-04-18T21:52:51.043 INFO:tasks.rados.rados.0.smithi097.stdout:306: seq_num 121 ranges {0=491520,491520=802816,1294336=720896,2015232=32768} 2024-04-18T21:52:51.045 INFO:tasks.rados.rados.0.smithi097.stdout:306: writing smithi09731216-49 from 0 to 491520 tid 1 2024-04-18T21:52:51.050 INFO:tasks.rados.rados.0.smithi097.stdout:306: writing smithi09731216-49 from 491520 to 1294336 tid 2 2024-04-18T21:52:51.055 INFO:tasks.rados.rados.0.smithi097.stdout:306: writing smithi09731216-49 from 1294336 to 2015232 tid 3 2024-04-18T21:52:51.055 INFO:tasks.rados.rados.0.smithi097.stdout:306: writing smithi09731216-49 from 2015232 to 2048000 tid 4 2024-04-18T21:52:51.055 INFO:tasks.rados.rados.0.smithi097.stdout:307: read oid 48 snap -1 2024-04-18T21:52:51.055 INFO:tasks.rados.rados.0.smithi097.stdout:307: expect (ObjNum 98 snap 19 seq_num 98) 2024-04-18T21:52:51.055 INFO:tasks.rados.rados.0.smithi097.stdout:308: rollback oid 3 current snap is 23 2024-04-18T21:52:51.055 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 3 to 21 2024-04-18T21:52:51.055 INFO:tasks.rados.rados.0.smithi097.stdout:309: snap_create 2024-04-18T21:52:51.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:51.087 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:51.082+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:51.188 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:51.182+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=7, reqid=client.4509.0:94, at_version=30'5, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.571630705000004, 2024-04-18T21:52:51.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201169337, 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661074+0000", 2024-04-18T21:52:51.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661077+0000", 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4600000000000002e-06 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661394+0000", 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031720499999999998 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661424+0000", 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9822999999999999e-05 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661441+0000", 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7496e-05 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661661+0000", 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000219602 2024-04-18T21:52:51.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661676+0000", 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5333e-05 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862187+0000", 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200510299 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862244+0000", 2024-04-18T21:52:51.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7119e-05 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.571227398000005, 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201435581, 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661478+0000", 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661479+0000", 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0839999999999999e-06 2024-04-18T21:52:51.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661509+0000", 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0389999999999999e-05 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661529+0000", 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0219000000000001e-05 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.661553+0000", 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3836999999999999e-05 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861851+0000", 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20029786299999999 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.861866+0000", 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4769999999999999e-05 2024-04-18T21:52:51.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862836+0000", 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097006600000000001 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.862913+0000", 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7352000000000001e-05 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.14s6 32/31 ECSubWrite(tid=20, reqid=client.4509.0:125, at_version=32'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 62.636200387999999, 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20154013100000001, 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596505+0000", 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3799999999999997e-07 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596521+0000", 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5455999999999999e-05 2024-04-18T21:52:51.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.596532+0000", 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0881e-05 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.696606+0000", 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100073932 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796760+0000", 2024-04-18T21:52:51.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10015476600000001 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.796771+0000", 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0278999999999999e-05 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.797921+0000", 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001150657 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.798045+0000", 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012362199999999999 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es6 41/26 ECSubWrite(tid=30, reqid=client.4509.0:316, at_version=41'18, trim_to=0'0, roll_forward_to=38'15))", 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 53.036804533000002, 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200900518, 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.247 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195901+0000", 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195902+0000", 2024-04-18T21:52:51.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3699999999999999e-07 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195937+0000", 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5682000000000002e-05 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.195960+0000", 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2940999999999999e-05 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296055+0000", 2024-04-18T21:52:51.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100094687 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296167+0000", 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011265900000000001 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.296183+0000", 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5605000000000001e-05 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.396721+0000", 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10053751700000001 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.396801+0000", 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0489999999999997e-05 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.251 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.17s4,RELEASE,e:42,from:8(0),reservation_nonce:0)", 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.445441064000001, 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20038272400000001, 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787264+0000", 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787265+0000", 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.02e-06 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787274+0000", 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.439e-06 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787295+0000", 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1121e-05 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.787312+0000", 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7838000000000001e-05 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987569+0000", 2024-04-18T21:52:51.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200256506 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987603+0000", 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4072000000000002e-05 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:59.987647+0000", 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3727999999999999e-05 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.255 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.11s3 47/45 ECSubRead(tid=45, to_read={3:8fad7925:::smithi09731216-43:head=0,262144,0}, subchunks={3:8fad7925:::smithi09731216-43:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 47.834968435999997, 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20153923400000001, 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397737+0000", 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397739+0000", 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.367e-06 2024-04-18T21:52:51.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397761+0000", 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1525000000000001e-05 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.397804+0000", 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3488000000000001e-05 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.497972+0000", 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100167611 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.598288+0000", 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100316505 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:03.599276+0000", 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00098773799999999994 2024-04-18T21:52:51.259 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpReadReply(3.12s0 52/51 ECSubReadReply(tid=0, attrs_read=0))", 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 42.245988070999999, 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20394542500000001, 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986717+0000", 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:08.986718+0000", 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087949+0000", 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10123096600000001 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.087978+0000", 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9116999999999999e-05 2024-04-18T21:52:51.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.188141+0000", 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10016269799999999 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.190504+0000", 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023637250000000001 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:09.190662+0000", 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000158076 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 64/58 ECSubWrite(tid=206, reqid=osd.7.0:206, at_version=64'51, trim_to=0'0, roll_forward_to=62'47))", 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.492846+0000", 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.739859515999999, 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20166795100000001, 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.264 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492846+0000", 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492846+0000", 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492846+0000", 2024-04-18T21:52:51.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9100000000000003e-07 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492878+0000", 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1483999999999999e-05 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.492892+0000", 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4476e-05 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.592975+0000", 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100083016 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.693140+0000", 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100164527 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.693156+0000", 2024-04-18T21:52:51.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6273000000000002e-05 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.694448+0000", 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001292039 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.694514+0000", 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5345000000000005e-05 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.268 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 64/58 ECSubWrite(tid=208, reqid=osd.7.0:206, at_version=0'0, trim_to=0'0, roll_forward_to=64'51))", 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.885368+0000", 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 30.347337503999999, 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20183865500000001, 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885368+0000", 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885368+0000", 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885369+0000", 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3340000000000001e-06 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885408+0000", 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9511000000000002e-05 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.885436+0000", 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7804999999999999e-05 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.985514+0000", 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100078234 2024-04-18T21:52:51.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.985605+0000", 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0080999999999994e-05 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.985617+0000", 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.27e-05 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.087123+0000", 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101505472 2024-04-18T21:52:51.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.087206+0000", 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3517999999999996e-05 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 68/58 ECSubWrite(tid=215, reqid=osd.7.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.920355+0000", 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.312350521999999, 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20283171999999999, 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.273 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920355+0000", 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920355+0000", 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920357+0000", 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3659999999999999e-06 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920382+0000", 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5085000000000001e-05 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920413+0000", 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1105000000000001e-05 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920430+0000", 2024-04-18T21:52:51.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6668999999999999e-05 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.022477+0000", 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102047579 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.022493+0000", 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5625999999999999e-05 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.123112+0000", 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100619477 2024-04-18T21:52:51.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.123186+0000", 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3813000000000006e-05 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDScrubReserve(3.13s7,QREQUEST,e:68,from:7(0),reservation_nonce:7)", 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:25.324753+0000", 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 25.907952444999999, 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20066576, 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.277 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324753+0000", 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324753+0000", 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324755+0000", 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0269999999999998e-06 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324764+0000", 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0219999999999993e-06 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.324794+0000", 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9802e-05 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.424947+0000", 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10015381900000001 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.525281+0000", 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10033357 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.525321+0000", 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0108000000000003e-05 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.525418+0000", 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7411999999999995e-05 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 73/58 ECSubWrite(tid=218, reqid=client.4509.0:563, at_version=73'59, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:51.281 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 20.807733316, 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30546464400000001, 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424974+0000", 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9e-06 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425568+0000", 2024-04-18T21:52:51.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00059416700000000005 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425597+0000", 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9305000000000001e-05 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425613+0000", 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5449000000000001e-05 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.625991+0000", 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200377891 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.626011+0000", 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0956000000000001e-05 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730378+0000", 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104366235 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730436+0000", 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8740999999999999e-05 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 73/58 ECSubWrite(tid=219, reqid=client.4509.0:564, at_version=73'60, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 20.797394190999999, 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29521209100000001, 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.286 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435313+0000", 2024-04-18T21:52:51.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.807e-06 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435737+0000", 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042456200000000002 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435778+0000", 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0555e-05 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.536874+0000", 2024-04-18T21:52:51.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101095886 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.727552+0000", 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19067846699999999 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.727571+0000", 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8941999999999999e-05 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730493+0000", 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029220800000000001 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730523+0000", 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9791999999999999e-05 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.as2 74/26 ECSubWrite(tid=61, reqid=client.4509.0:572, at_version=74'17, trim_to=0'0, roll_forward_to=70'15))", 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.639729+0000", 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 20.592975868, 2024-04-18T21:52:51.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200655103, 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.639729+0000", 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.639729+0000", 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.639731+0000", 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.866e-06 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.640108+0000", 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00037644799999999998 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.640135+0000", 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7212e-05 2024-04-18T21:52:51.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.640152+0000", 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6943e-05 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.736587+0000", 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.096435114000000002 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.736603+0000", 2024-04-18T21:52:51.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5719e-05 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.840260+0000", 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10365738400000001 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.840384+0000", 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012441700000000001 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.294 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(3.bs3 93/90 [PushOp(3:d4cd3926:::smithi09731216-39:18, version: 91'51, data_included: [0~167936], data_size: 167936, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:d4cd3926:::smithi09731216-39:18@91'51, size: 671744, copy_subset: [], clone_subset: {}, snapset: 18=[]:{18: [17, 15, 10, a]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 671744, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.417806+0000", 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.81489925900000004, 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20222009099999999, 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.417806+0000", 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.417806+0000", 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.417807+0000", 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.412e-06 2024-04-18T21:52:51.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.418009+0000", 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020197699999999999 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.418035+0000", 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5437999999999999e-05 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.518124+0000", 2024-04-18T21:52:51.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100089332 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.618405+0000", 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10028096 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.620026+0000", 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001620972 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.298 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:731 3.12s0 3:4c1ede14:::smithi09731216-49:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.487911+0000", 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.74479450599999997, 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.316265252, 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 731 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487911+0000", 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487911+0000", 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487913+0000", 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2919999999999998e-06 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487921+0000", 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4749999999999993e-06 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487946+0000", 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4958999999999999e-05 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487958+0000", 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1948e-05 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488076+0000", 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117897 2024-04-18T21:52:51.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488092+0000", 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5722e-05 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.498251+0000", 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010159559 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.498408+0000", 2024-04-18T21:52:51.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015634399999999999 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.498960+0000", 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00055209400000000004 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.503027+0000", 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040673569999999997 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.803500+0000", 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30047284800000001 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.804176+0000", 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067575700000000003 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:732 3.12s0 3:4c1ede14:::smithi09731216-49:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:51.305 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.744716669, 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.417434898, 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 732 2024-04-18T21:52:51.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487989+0000", 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8299999999999997e-07 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487992+0000", 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.253e-06 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488000+0000", 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7090000000000002e-06 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488006+0000", 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3550000000000001e-06 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488154+0000", 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014786700000000001 2024-04-18T21:52:51.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488167+0000", 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2836e-05 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.499213+0000", 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011045981999999999 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.499412+0000", 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000198519 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.500214+0000", 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00080208099999999995 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.503072+0000", 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00285819 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.904913+0000", 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40184145999999998 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.905423+0000", 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00051006300000000005 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.312 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=22, last_complete=94'21, committed=1, applied=1))", 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.702823+0000", 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.52988254300000004, 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20169795400000001, 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702823+0000", 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702823+0000", 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702823+0000", 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.53e-07 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702826+0000", 2024-04-18T21:52:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.18e-06 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702835+0000", 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8000000000000004e-06 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702844+0000", 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2089999999999994e-06 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.904422+0000", 2024-04-18T21:52:51.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201577955 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.904521+0000", 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8456999999999996e-05 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=22, last_complete=64'13, committed=1, applied=1))", 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.702894+0000", 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.52981159799999999, 2024-04-18T21:52:51.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201840723, 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702894+0000", 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702894+0000", 2024-04-18T21:52:51.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702894+0000", 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.53e-07 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702897+0000", 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8270000000000002e-06 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702912+0000", 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5172e-05 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.702946+0000", 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3958999999999998e-05 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.904646+0000", 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20170020499999999 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.904734+0000", 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8207000000000004e-05 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s6 94/88 ECSubWrite(tid=83, reqid=osd.1.0:82, at_version=0'0, trim_to=0'0, roll_forward_to=94'46))", 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.710166+0000", 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.52253955399999996, 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20138810300000001, 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.320 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710166+0000", 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710166+0000", 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710167+0000", 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7570000000000001e-06 2024-04-18T21:52:51.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710187+0000", 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0102000000000002e-05 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710214+0000", 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7056e-05 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.710238+0000", 2024-04-18T21:52:51.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3093000000000001e-05 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.810421+0000", 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10018368 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.810430+0000", 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5210000000000002e-06 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.911524+0000", 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10109444400000001 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.911554+0000", 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9450000000000001e-05 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.324 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:51.326 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:52:51.327 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:52:51.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:51.329 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-04-18T21:52:50.485542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91390496,"num_objects":100,"num_object_clones":51,"num_object_copies":788,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":34,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":8,"num_read":333,"num_read_kb":80329,"num_write":565,"num_write_kb":71658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":95305728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":897,"ondisk_log_size":897,"up":210,"acting":209,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":206,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":943718400,"kb_used":530792,"kb_used_data":193992,"kb_used_omap":15,"kb_used_meta":336624,"kb_avail":943187608,"statfs":{"total":966367641600,"available":965824110592,"internally_reserved":0,"allocated":198647808,"data_stored":252742117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15954,"internal_metadata":344703406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,3,18,0,0,0,5,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5996544,"num_objects":2,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":23,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":-1,"num_read":17,"num_read_kb":9943,"num_write":99,"num_write_kb":7936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":14003680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.060600"},"pg_stats":[{"pgid":"3.17","version":"51'8","reported_seq":296,"reported_epoch":92,"state":"remapped+peering","last_fresh":"2024-04-18T21:52:48.475007+0000","last_change":"2024-04-18T21:52:48.475007+0000","last_active":"2024-04-18T21:52:44.524289+0000","last_peered":"2024-04-18T21:52:48.385748+0000","last_clean":"2024-04-18T21:52:43.426361+0000","last_became_active":"2024-04-18T21:51:48.483038+0000","last_became_peered":"2024-04-18T21:52:47.836528+0000","last_unstale":"2024-04-18T21:52:48.475007+0000","last_undegraded":"2024-04-18T21:52:48.475007+0000","last_fullsized":"2024-04-18T21:52:48.475007+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'6","last_scrub_stamp":"2024-04-18T21:51:59.786488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:59.786488+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0.001588428,"stat_sum":{"num_bytes":671744,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,10,9,1,4,5],"acting":[8,3,2147483647,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.16","version":"81'14","reported_seq":278,"reported_epoch":92,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-18T21:52:48.703525+0000","last_change":"2024-04-18T21:52:48.063827+0000","last_active":"2024-04-18T21:52:48.703525+0000","last_peered":"2024-04-18T21:52:48.703525+0000","last_clean":"2024-04-18T21:52:38.103206+0000","last_became_active":"2024-04-18T21:52:47.859287+0000","last_became_peered":"2024-04-18T21:52:47.859287+0000","last_unstale":"2024-04-18T21:52:48.703525+0000","last_undegraded":"2024-04-18T21:52:47.435283+0000","last_fullsized":"2024-04-18T21:52:47.432320+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'3","last_scrub_stamp":"2024-04-18T21:52:01.391703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.391703+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0.39815574799999998,"stat_sum":{"num_bytes":2146304,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":6290,"num_write":4,"num_write_kb":1504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,6,10,2,1,5],"acting":[11,3,4,2147483647,10,2147483647,1,5],"avail_no_missing":["11(0)","1(6)","2(0)","3(1)","4(2)","5(7)","6(5)","7(3)","10(4)"],"object_location_counts":[{"shards":"1(6),2(0),3(1),4(2),5(7),6(5),7(3),10(4),11(0)","objects":2}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.15","version":"92'11","reported_seq":335,"reported_epoch":93,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-18T21:52:49.582582+0000","last_change":"2024-04-18T21:52:48.267669+0000","last_active":"2024-04-18T21:52:49.582582+0000","last_peered":"2024-04-18T21:52:49.582582+0000","last_clean":"2024-04-18T21:52:44.429656+0000","last_became_active":"2024-04-18T21:52:48.169208+0000","last_became_peered":"2024-04-18T21:52:48.169208+0000","last_unstale":"2024-04-18T21:52:49.582582+0000","last_undegraded":"2024-04-18T21:52:47.554041+0000","last_fullsized":"2024-04-18T21:52:47.552400+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.079751002000000001,"stat_sum":{"num_bytes":425984,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":425984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,4,11,3,9],"acting":[5,8,10,6,2147483647,11,3,9],"avail_no_missing":["5(0)","3(6)","6(3)","7(4)","8(1)","9(7)","10(2)","11(5)"],"object_location_counts":[{"shards":"3(6),5(0),6(3),8(1),9(7),10(2),11(5)","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.14","version":"51'8","reported_seq":187,"reported_epoch":89,"state":"remapped+peering","last_fresh":"2024-04-18T21:52:45.426427+0000","last_change":"2024-04-18T21:52:46.430210+0000","last_active":"2024-04-18T21:52:45.426427+0000","last_peered":"2024-04-18T21:52:45.161137+0000","last_clean":"2024-04-18T21:52:45.161137+0000","last_became_active":"2024-04-18T21:52:44.757426+0000","last_became_peered":"2024-04-18T21:52:44.757426+0000","last_unstale":"2024-04-18T21:52:45.426427+0000","last_undegraded":"2024-04-18T21:52:45.426427+0000","last_fullsized":"2024-04-18T21:52:45.426427+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1515,"objects_trimmed":0,"snaptrim_duration":0.012297201000000001,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,11,5,6,9,1],"acting":[4,2,10,11,2147483647,6,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,5,6,7,9,10,11],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-18T21:52:46.442080+0000","last_change":"2024-04-18T21:52:28.499393+0000","last_active":"2024-04-18T21:52:46.442080+0000","last_peered":"2024-04-18T21:52:46.442080+0000","last_clean":"2024-04-18T21:52:46.442080+0000","last_became_active":"2024-04-18T21:52:28.498662+0000","last_became_peered":"2024-04-18T21:52:28.498662+0000","last_unstale":"2024-04-18T21:52:46.442080+0000","last_undegraded":"2024-04-18T21:52:46.442080+0000","last_fullsized":"2024-04-18T21:52:46.442080+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'18","reported_seq":300,"reported_epoch":92,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-18T21:52:48.700746+0000","last_change":"2024-04-18T21:52:48.062891+0000","last_active":"2024-04-18T21:52:48.700746+0000","last_peered":"2024-04-18T21:52:48.700746+0000","last_clean":"2024-04-18T21:52:44.428739+0000","last_became_active":"2024-04-18T21:52:47.860008+0000","last_became_peered":"2024-04-18T21:52:47.860008+0000","last_unstale":"2024-04-18T21:52:48.700746+0000","last_undegraded":"2024-04-18T21:52:47.530859+0000","last_fullsized":"2024-04-18T21:52:47.528578+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.20600609,"stat_sum":{"num_bytes":589824,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,6,10,2,1,5],"acting":[11,3,4,2147483647,10,2147483647,1,5],"avail_no_missing":["11(0)","1(6)","3(1)","4(2)","5(7)","6(5)","7(3)","10(4)"],"object_location_counts":[{"shards":"1(6),3(1),4(2),5(7),6(5),7(3),10(4),11(0)","objects":3}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-18T21:52:49.483997+0000","last_change":"2024-04-18T21:52:28.588502+0000","last_active":"2024-04-18T21:52:49.483997+0000","last_peered":"2024-04-18T21:52:49.483997+0000","last_clean":"2024-04-18T21:52:49.483997+0000","last_became_active":"2024-04-18T21:52:28.587613+0000","last_became_peered":"2024-04-18T21:52:28.587613+0000","last_unstale":"2024-04-18T21:52:49.483997+0000","last_undegraded":"2024-04-18T21:52:49.483997+0000","last_fullsized":"2024-04-18T21:52:49.483997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:52.162125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":286,"reported_epoch":91,"state":"peering","last_fresh":"2024-04-18T21:52:47.542248+0000","last_change":"2024-04-18T21:52:48.474239+0000","last_active":"2024-04-18T21:52:44.525399+0000","last_peered":"2024-04-18T21:52:47.542248+0000","last_clean":"2024-04-18T21:52:47.542248+0000","last_became_active":"2024-04-18T21:51:48.380296+0000","last_became_peered":"2024-04-18T21:52:46.832868+0000","last_unstale":"2024-04-18T21:52:47.542248+0000","last_undegraded":"2024-04-18T21:52:47.542248+0000","last_fullsized":"2024-04-18T21:52:47.542248+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.001592191,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,10,9,1,4,5],"acting":[8,3,2,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3,4,5,9,10],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":151,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-18T21:52:49.583683+0000","last_change":"2024-04-18T21:51:33.299595+0000","last_active":"2024-04-18T21:52:49.583683+0000","last_peered":"2024-04-18T21:52:49.583683+0000","last_clean":"2024-04-18T21:52:49.583683+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:52:49.583683+0000","last_undegraded":"2024-04-18T21:52:49.583683+0000","last_fullsized":"2024-04-18T21:52:49.583683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"30'2","reported_seq":332,"reported_epoch":88,"state":"remapped+peering","last_fresh":"2024-04-18T21:52:44.530473+0000","last_change":"2024-04-18T21:52:45.530708+0000","last_active":"2024-04-18T21:52:45.530708+0000","last_peered":"2024-04-18T21:52:44.530473+0000","last_clean":"2024-04-18T21:52:38.575599+0000","last_became_active":"2024-04-18T21:52:42.006272+0000","last_became_peered":"2024-04-18T21:52:42.006272+0000","last_unstale":"2024-04-18T21:52:44.530473+0000","last_undegraded":"2024-04-18T21:52:44.530473+0000","last_fullsized":"2024-04-18T21:52:44.530473+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:51:50.311307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.311307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1203,"objects_trimmed":0,"snaptrim_duration":0.30169549200000001,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3170,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1081344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,6,9,11],"acting":[10,2,1,5,8,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,5,6,7,8,9,11],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-18T21:52:48.501852+0000","last_change":"2024-04-18T21:52:40.516658+0000","last_active":"2024-04-18T21:52:48.501852+0000","last_peered":"2024-04-18T21:52:48.501852+0000","last_clean":"2024-04-18T21:52:48.501852+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:52:48.501852+0000","last_undegraded":"2024-04-18T21:52:48.501852+0000","last_fullsized":"2024-04-18T21:52:48.501852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:59.295293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'34","reported_seq":363,"reported_epoch":93,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-18T21:52:49.499531+0000","last_change":"2024-04-18T21:52:49.767977+0000","last_active":"2024-04-18T21:52:49.499531+0000","last_peered":"2024-04-18T21:52:49.499531+0000","last_clean":"2024-04-18T21:52:44.530036+0000","last_became_active":"2024-04-18T21:52:47.966278+0000","last_became_peered":"2024-04-18T21:52:47.966278+0000","last_unstale":"2024-04-18T21:52:49.499531+0000","last_undegraded":"2024-04-18T21:52:47.450137+0000","last_fullsized":"2024-04-18T21:52:47.448183+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'7","last_scrub_stamp":"2024-04-18T21:51:51.017447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.017447+0000","objects_scrubbed":1,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2135,"objects_trimmed":0,"snaptrim_duration":0.28659757299999999,"stat_sum":{"num_bytes":4603904,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":30,"num_write_kb":3680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1802240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,4,11,3,9],"acting":[5,8,10,6,2147483647,11,3,9],"avail_no_missing":["5(0)","3(6)","6(3)","7(4)","8(1)","9(7)","10(2)","11(5)"],"object_location_counts":[{"shards":"3(6),5(0),6(3),7(4),8(1),9(7),10(2),11(5)","objects":3},{"shards":"3(6),5(0),6(3),8(1),9(7),10(2),11(5)","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.3","version":"89'140","reported_seq":538,"reported_epoch":90,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-18T21:52:46.434918+0000","last_change":"2024-04-18T21:52:43.020563+0000","last_active":"2024-04-18T21:52:46.434918+0000","last_peered":"2024-04-18T21:52:46.434918+0000","last_clean":"2024-04-18T21:52:27.421683+0000","last_became_active":"2024-04-18T21:52:41.805267+0000","last_became_peered":"2024-04-18T21:52:41.805267+0000","last_unstale":"2024-04-18T21:52:46.434918+0000","last_undegraded":"2024-04-18T21:52:41.486800+0000","last_fullsized":"2024-04-18T21:52:41.484515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.41560456099999998,"stat_sum":{"num_bytes":22380544,"num_objects":19,"num_object_clones":11,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":31,"num_read_kb":17290,"num_write":29,"num_write_kb":15472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":18825216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11,6,1,5,2,3],"acting":[4,2147483647,11,6,1,5,2,2147483647],"avail_no_missing":["4(0)","0(7)","1(4)","2(6)","3(1)","5(5)","6(3)","11(2)"],"object_location_counts":[{"shards":"1(4),2(6),4(0),5(5),6(3),11(2)","objects":18}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-18T21:52:49.581193+0000","last_change":"2024-04-18T21:51:33.011841+0000","last_active":"2024-04-18T21:52:49.581193+0000","last_peered":"2024-04-18T21:52:49.581193+0000","last_clean":"2024-04-18T21:52:49.581193+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:52:49.581193+0000","last_undegraded":"2024-04-18T21:52:49.581193+0000","last_fullsized":"2024-04-18T21:52:49.581193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:58.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"79'34","reported_seq":368,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-04-18T21:52:46.047116+0000","last_change":"2024-04-18T21:52:46.047116+0000","last_active":"2024-04-18T21:52:46.047116+0000","last_peered":"2024-04-18T21:52:46.047116+0000","last_clean":"2024-04-18T21:52:46.047116+0000","last_became_active":"2024-04-18T21:52:45.939975+0000","last_became_peered":"2024-04-18T21:52:45.939975+0000","last_unstale":"2024-04-18T21:52:46.047116+0000","last_undegraded":"2024-04-18T21:52:46.047116+0000","last_fullsized":"2024-04-18T21:52:46.047116+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:27.029581+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.48276140699999998,"stat_sum":{"num_bytes":4046848,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":15,"num_read_kb":4611,"num_write":3,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5373952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-18T21:52:49.585683+0000","last_change":"2024-04-18T21:52:40.415249+0000","last_active":"2024-04-18T21:52:49.585683+0000","last_peered":"2024-04-18T21:52:49.585683+0000","last_clean":"2024-04-18T21:52:49.585683+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:52:49.585683+0000","last_undegraded":"2024-04-18T21:52:49.585683+0000","last_fullsized":"2024-04-18T21:52:49.585683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:52:50.415123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"92'24","reported_seq":302,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-18T21:52:48.853806+0000","last_change":"2024-04-18T21:52:48.956858+0000","last_active":"2024-04-18T21:52:48.853806+0000","last_peered":"2024-04-18T21:52:48.853806+0000","last_clean":"2024-04-18T21:52:48.853806+0000","last_became_active":"2024-04-18T21:52:44.644662+0000","last_became_peered":"2024-04-18T21:52:44.644662+0000","last_unstale":"2024-04-18T21:52:48.853806+0000","last_undegraded":"2024-04-18T21:52:48.853806+0000","last_fullsized":"2024-04-18T21:52:48.853806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'6","last_scrub_stamp":"2024-04-18T21:51:50.749104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:50.749104+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:14.590326+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0.40866950299999999,"stat_sum":{"num_bytes":6881280,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":530,"num_write":47,"num_write_kb":6544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"3"},{"start":"18","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":3,"reported_epoch":89,"state":"peering","last_fresh":"2024-04-18T21:52:45.425711+0000","last_change":"2024-04-18T21:52:45.425711+0000","last_active":"2024-04-18T21:51:31.787097+0000","last_peered":"2024-04-18T21:51:31.787097+0000","last_clean":"2024-04-18T21:51:31.787097+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:52:45.425711+0000","last_undegraded":"2024-04-18T21:52:45.425711+0000","last_fullsized":"2024-04-18T21:52:45.425711+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"85'23","reported_seq":320,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-18T21:52:45.536853+0000","last_change":"2024-04-18T21:52:45.537997+0000","last_active":"2024-04-18T21:52:45.536853+0000","last_peered":"2024-04-18T21:52:45.536853+0000","last_clean":"2024-04-18T21:52:45.536853+0000","last_became_active":"2024-04-18T21:52:41.694736+0000","last_became_peered":"2024-04-18T21:52:41.694736+0000","last_unstale":"2024-04-18T21:52:45.536853+0000","last_undegraded":"2024-04-18T21:52:45.536853+0000","last_fullsized":"2024-04-18T21:52:45.536853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:27.656739+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.001647452,"stat_sum":{"num_bytes":1605632,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"},{"start":"16","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-18T21:52:49.615092+0000","last_change":"2024-04-18T21:52:40.514599+0000","last_active":"2024-04-18T21:52:49.615092+0000","last_peered":"2024-04-18T21:52:49.615092+0000","last_clean":"2024-04-18T21:52:49.615092+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:52:49.615092+0000","last_undegraded":"2024-04-18T21:52:49.615092+0000","last_fullsized":"2024-04-18T21:52:49.615092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":291,"reported_epoch":90,"state":"remapped+peering","last_fresh":"2024-04-18T21:52:46.545633+0000","last_change":"2024-04-18T21:52:46.545633+0000","last_active":"2024-04-18T21:52:27.394239+0000","last_peered":"2024-04-18T21:52:26.415071+0000","last_clean":"2024-04-18T21:52:26.415071+0000","last_became_active":"2024-04-18T21:51:30.078132+0000","last_became_peered":"2024-04-18T21:51:30.078132+0000","last_unstale":"2024-04-18T21:52:46.545633+0000","last_undegraded":"2024-04-18T21:52:46.545633+0000","last_fullsized":"2024-04-18T21:52:46.545633+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:51:28.758641+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"93'28","reported_seq":338,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-18T21:52:49.888604+0000","last_change":"2024-04-18T21:52:49.890171+0000","last_active":"2024-04-18T21:52:49.888604+0000","last_peered":"2024-04-18T21:52:49.888604+0000","last_clean":"2024-04-18T21:52:49.888604+0000","last_became_active":"2024-04-18T21:52:43.857309+0000","last_became_peered":"2024-04-18T21:52:43.857309+0000","last_unstale":"2024-04-18T21:52:49.888604+0000","last_undegraded":"2024-04-18T21:52:49.888604+0000","last_fullsized":"2024-04-18T21:52:49.888604+0000","mapping_epoch":74,"log_start":"81'26","ondisk_log_start":"81'26","created":26,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":28,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:13.836755+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.40818402100000001,"stat_sum":{"num_bytes":2523136,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2258,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3506176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,6,1],"acting":[3,10,2,4,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"a","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"3"},{"start":"18","length":"1"}]},{"pgid":"3.9","version":"83'30","reported_seq":336,"reported_epoch":93,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-18T21:52:49.583366+0000","last_change":"2024-04-18T21:52:48.654275+0000","last_active":"2024-04-18T21:52:49.583366+0000","last_peered":"2024-04-18T21:52:49.583366+0000","last_clean":"2024-04-18T21:52:44.521893+0000","last_became_active":"2024-04-18T21:52:48.237783+0000","last_became_peered":"2024-04-18T21:52:48.237783+0000","last_unstale":"2024-04-18T21:52:49.583366+0000","last_undegraded":"2024-04-18T21:52:49.583366+0000","last_fullsized":"2024-04-18T21:52:49.583366+0000","mapping_epoch":90,"log_start":"64'24","ondisk_log_start":"64'24","created":26,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":26,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.77944305899999999,"stat_sum":{"num_bytes":1409024,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":98,"num_write":6,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,1,5,9,4,10],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":["3(0)","1(3)","2(1)","4(6)","5(4)","6(7)","7(2)","9(5)"],"object_location_counts":[{"shards":"1(3),2(1),3(0),4(6),5(4),6(7),7(2),9(5)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.a","version":"81'30","reported_seq":290,"reported_epoch":89,"state":"remapped+peering","last_fresh":"2024-04-18T21:52:45.439578+0000","last_change":"2024-04-18T21:52:46.447201+0000","last_active":"2024-04-18T21:52:45.439578+0000","last_peered":"2024-04-18T21:52:44.433810+0000","last_clean":"2024-04-18T21:52:44.433810+0000","last_became_active":"2024-04-18T21:51:43.266273+0000","last_became_peered":"2024-04-18T21:51:43.266273+0000","last_unstale":"2024-04-18T21:52:45.439578+0000","last_undegraded":"2024-04-18T21:52:45.439578+0000","last_fullsized":"2024-04-18T21:52:45.439578+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.39383959099999999,"stat_sum":{"num_bytes":3260416,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":3269,"num_write":13,"num_write_kb":3296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,10],"acting":[6,4,9,2,3,1,8,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3,4,7,8,9,10],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.b","version":"81'50","reported_seq":315,"reported_epoch":88,"state":"peering","last_fresh":"2024-04-18T21:52:44.528185+0000","last_change":"2024-04-18T21:52:45.435521+0000","last_active":"2024-04-18T21:52:45.435521+0000","last_peered":"2024-04-18T21:52:44.528185+0000","last_clean":"2024-04-18T21:52:44.528185+0000","last_became_active":"2024-04-18T21:51:43.142440+0000","last_became_peered":"2024-04-18T21:51:43.142440+0000","last_unstale":"2024-04-18T21:52:44.528185+0000","last_undegraded":"2024-04-18T21:52:44.528185+0000","last_fullsized":"2024-04-18T21:52:44.528185+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.49866226600000002,"stat_sum":{"num_bytes":4833280,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":13017,"num_write":20,"num_write_kb":6224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,9,1,2,8,6],"acting":[10,3,4,9,1,2,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3,4,6,7,8,9],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.c","version":"93'77","reported_seq":441,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2024-04-18T21:52:50.102512+0000","last_change":"2024-04-18T21:52:49.682828+0000","last_active":"2024-04-18T21:52:50.102512+0000","last_peered":"2024-04-18T21:52:50.102512+0000","last_clean":"2024-04-18T21:52:50.102512+0000","last_became_active":"2024-04-18T21:52:41.700367+0000","last_became_peered":"2024-04-18T21:52:41.700367+0000","last_unstale":"2024-04-18T21:52:50.102512+0000","last_undegraded":"2024-04-18T21:52:50.102512+0000","last_fullsized":"2024-04-18T21:52:50.102512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:11.588017+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0043595580000000004,"stat_sum":{"num_bytes":4292608,"num_objects":7,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":16796,"num_write":67,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":9289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,11,10,9,2],"acting":[5,8,6,1,11,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"3"},{"start":"18","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":2,"reported_epoch":89,"state":"peering","last_fresh":"2024-04-18T21:51:42.913953+0000","last_change":"2024-04-18T21:52:46.111767+0000","last_active":"2024-04-18T21:51:41.891788+0000","last_peered":"2024-04-18T21:51:41.891788+0000","last_clean":"2024-04-18T21:51:41.891788+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T21:51:42.913953+0000","last_undegraded":"2024-04-18T21:51:42.913953+0000","last_fullsized":"2024-04-18T21:51:42.913953+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1,6,3,8,4,2],"acting":[10,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.e","version":"88'62","reported_seq":406,"reported_epoch":93,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-18T21:52:49.883678+0000","last_change":"2024-04-18T21:52:48.392903+0000","last_active":"2024-04-18T21:52:49.883678+0000","last_peered":"2024-04-18T21:52:49.883678+0000","last_clean":"2024-04-18T21:52:44.945334+0000","last_became_active":"2024-04-18T21:52:47.857146+0000","last_became_peered":"2024-04-18T21:52:47.857146+0000","last_unstale":"2024-04-18T21:52:49.883678+0000","last_undegraded":"2024-04-18T21:52:49.883678+0000","last_fullsized":"2024-04-18T21:52:47.434722+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.88469175600000005,"stat_sum":{"num_bytes":8060928,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":4235,"num_write":31,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":15122432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1,4,8,11,9,5],"acting":[2,6,1,4,8,11,9,2147483647],"avail_no_missing":["2(0)","1(2)","4(3)","6(1)","7(7)","8(4)","9(6)","11(5)"],"object_location_counts":[{"shards":"1(2),2(0),4(3),6(1),7(7),8(4),9(6),11(5)","objects":2},{"shards":"1(2),2(0),4(3),6(1),8(4),9(6),11(5)","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.f","version":"92'35","reported_seq":33,"reported_epoch":93,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-18T21:52:49.677751+0000","last_change":"2024-04-18T21:52:48.992101+0000","last_active":"2024-04-18T21:52:49.677751+0000","last_peered":"2024-04-18T21:52:49.677751+0000","last_clean":"2024-04-18T21:52:42.605471+0000","last_became_active":"2024-04-18T21:52:47.845423+0000","last_became_peered":"2024-04-18T21:52:47.845423+0000","last_unstale":"2024-04-18T21:52:49.677751+0000","last_undegraded":"2024-04-18T21:52:49.677751+0000","last_fullsized":"2024-04-18T21:52:47.427048+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063621000000000001,"stat_sum":{"num_bytes":5783552,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":2888,"num_write":42,"num_write_kb":6384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":18382848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,10,4,2,5,11,8],"acting":[3,1,10,2147483647,2,5,11,8],"avail_no_missing":["1(1)","2(4)","3(3)","5(5)","7(0)","8(7)","10(2)","11(6)"],"object_location_counts":[{"shards":"1(1),2(4),3(0),5(5),8(7),10(2),11(6)","objects":2},{"shards":"1(1),2(4),3(3),5(5),7(0),8(7),10(2),11(6)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.10","version":"87'56","reported_seq":352,"reported_epoch":89,"state":"remapped+peering","last_fresh":"2024-04-18T21:52:45.531028+0000","last_change":"2024-04-18T21:52:46.528728+0000","last_active":"2024-04-18T21:52:45.531028+0000","last_peered":"2024-04-18T21:52:44.433185+0000","last_clean":"2024-04-18T21:52:44.433185+0000","last_became_active":"2024-04-18T21:52:41.688684+0000","last_became_peered":"2024-04-18T21:52:41.688684+0000","last_unstale":"2024-04-18T21:52:45.531028+0000","last_undegraded":"2024-04-18T21:52:45.531028+0000","last_fullsized":"2024-04-18T21:52:45.531028+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":763,"objects_trimmed":1,"snaptrim_duration":0.59343419100000006,"stat_sum":{"num_bytes":6176768,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12992512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,11,8,4,10,3],"acting":[6,2,5,11,8,4,10,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,4,5,7,8,10,11],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.11","version":"81'34","reported_seq":277,"reported_epoch":93,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-18T21:52:49.482427+0000","last_change":"2024-04-18T21:52:48.650204+0000","last_active":"2024-04-18T21:52:49.482427+0000","last_peered":"2024-04-18T21:52:49.482427+0000","last_clean":"2024-04-18T21:52:44.731015+0000","last_became_active":"2024-04-18T21:52:48.242753+0000","last_became_peered":"2024-04-18T21:52:48.242753+0000","last_unstale":"2024-04-18T21:52:49.482427+0000","last_undegraded":"2024-04-18T21:52:47.527950+0000","last_fullsized":"2024-04-18T21:52:47.527031+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'24","last_scrub_stamp":"2024-04-18T21:52:12.154768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:12.154768+0000","objects_scrubbed":5,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1591,"objects_trimmed":1,"snaptrim_duration":0.37187643199999998,"stat_sum":{"num_bytes":3899392,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":5761,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,8],"acting":[3,4,6,10,1,5,2,2147483647],"avail_no_missing":["3(0)","1(4)","2(6)","4(1)","5(5)","6(2)","7(7)","10(3)"],"object_location_counts":[{"shards":"1(4),2(6),3(0),4(1),5(5),6(2),7(7),10(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.12","version":"91'19","reported_seq":290,"reported_epoch":93,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-18T21:52:49.484274+0000","last_change":"2024-04-18T21:52:47.856975+0000","last_active":"2024-04-18T21:52:49.484274+0000","last_peered":"2024-04-18T21:52:49.484274+0000","last_clean":"2024-04-18T21:52:39.094452+0000","last_became_active":"2024-04-18T21:52:47.847633+0000","last_became_peered":"2024-04-18T21:52:47.847633+0000","last_unstale":"2024-04-18T21:52:49.484274+0000","last_undegraded":"2024-04-18T21:52:47.444069+0000","last_fullsized":"2024-04-18T21:52:47.440629+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'11","last_scrub_stamp":"2024-04-18T21:52:15.060778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:15.060778+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":799,"objects_trimmed":0,"snaptrim_duration":0.30935977100000001,"stat_sum":{"num_bytes":2686976,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1638400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8,10,11,5,6,1],"acting":[9,2147483647,8,10,4,5,6,1],"avail_no_missing":["9(0)","1(7)","4(4)","5(5)","6(6)","7(1)","8(2)","10(3)"],"object_location_counts":[{"shards":"1(7),4(4),5(5),6(6),7(1),8(2),9(0),10(3)","objects":1},{"shards":"1(7),4(4),5(5),6(6),8(2),9(0),10(3)","objects":1}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.13","version":"88'73","reported_seq":343,"reported_epoch":93,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-18T21:52:49.683833+0000","last_change":"2024-04-18T21:52:47.950462+0000","last_active":"2024-04-18T21:52:49.683833+0000","last_peered":"2024-04-18T21:52:49.683833+0000","last_clean":"2024-04-18T21:52:45.338712+0000","last_became_active":"2024-04-18T21:52:47.847047+0000","last_became_peered":"2024-04-18T21:52:47.847047+0000","last_unstale":"2024-04-18T21:52:49.683833+0000","last_undegraded":"2024-04-18T21:52:47.535176+0000","last_fullsized":"2024-04-18T21:52:47.532367+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":5,"last_scrub":"68'57","last_scrub_stamp":"2024-04-18T21:52:26.089886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:26.089886+0000","objects_scrubbed":7,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":979,"objects_trimmed":1,"snaptrim_duration":0.31906991200000001,"stat_sum":{"num_bytes":3915776,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":25,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,8,6,11,1,9],"acting":[2147483647,5,4,8,6,11,1,9],"avail_no_missing":["5(1)","1(6)","4(2)","6(4)","7(0)","8(3)","9(7)","11(5)"],"object_location_counts":[{"shards":"1(6),4(2),5(1),6(4),7(0),8(3),9(7),11(5)","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":24,"stat_sum":{"num_bytes":90800128,"num_objects":98,"num_object_clones":51,"num_object_copies":784,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":34,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":8,"num_read":207,"num_read_kb":80220,"num_write":384,"num_write_kb":68720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":95305728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204705792,"data_stored":272326656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":1,"num_store_stats":3}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378643,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45776,"kb_used_data":10432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94326064,"statfs":{"total":96636764160,"available":96589889536,"internally_reserved":0,"allocated":10682368,"data_stored":12972197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378643,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57956,"kb_used_data":22608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94313884,"statfs":{"total":96636764160,"available":96577417216,"internally_reserved":0,"allocated":23150592,"data_stored":27589463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378644,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52772,"kb_used_data":17428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319068,"statfs":{"total":96636764160,"available":96582725632,"internally_reserved":0,"allocated":17846272,"data_stored":22655141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378643,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53012,"kb_used_data":17668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318828,"statfs":{"total":96636764160,"available":96582479872,"internally_reserved":0,"allocated":18092032,"data_stored":23404709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378644,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59840,"kb_used_data":24496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94312000,"statfs":{"total":96636764160,"available":96575488000,"internally_reserved":0,"allocated":25083904,"data_stored":30634333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411347,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51200,"kb_used_data":20012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320640,"statfs":{"total":96636764160,"available":96584335360,"internally_reserved":0,"allocated":20492288,"data_stored":25761251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":0,"seq":77309411347,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411347,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56888,"kb_used_data":21548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314952,"statfs":{"total":96636764160,"available":96578510848,"internally_reserved":0,"allocated":22065152,"data_stored":29771235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411348,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52748,"kb_used_data":21564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319092,"statfs":{"total":96636764160,"available":96582750208,"internally_reserved":0,"allocated":22081536,"data_stored":29815133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":77309411346,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50536,"kb_used_data":19356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321304,"statfs":{"total":96636764160,"available":96585015296,"internally_reserved":0,"allocated":19820544,"data_stored":26094786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,1,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444051,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50064,"kb_used_data":18880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321776,"statfs":{"total":96636764160,"available":96585498624,"internally_reserved":0,"allocated":19333120,"data_stored":24043869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3923968,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17477632,"data_stored":23318528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21569536,"data_stored":29573120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24375296,"data_stored":30384128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17235968,"data_stored":24199168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20791296,"data_stored":28962816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21516288,"data_stored":27942912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11321344,"data_stored":14819328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":18198528,"data_stored":23687168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":19341312,"data_stored":25075712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9596928,"data_stored":12324864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":19357696,"data_stored":26988544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:52:51.331 INFO:tasks.thrashosds.thrasher:Setting 3.2 to [2, 6, 0, 9, 11, 8, 1, 4] 2024-04-18T21:52:51.331 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.2', '2', '6', '0', '9', '11', '8', '1', '4'] 2024-04-18T21:52:51.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.2 2 6 0 9 11 8 1 4 2024-04-18T21:52:51.332 INFO:tasks.rados.rados.0.smithi097.stdout:303: finishing write tid 1 to smithi09731216-11 2024-04-18T21:52:51.332 INFO:tasks.rados.rados.0.smithi097.stdout:298: finishing write tid 1 to smithi09731216-35 2024-04-18T21:52:51.332 INFO:tasks.rados.rados.0.smithi097.stdout:298: finishing write tid 2 to smithi09731216-35 2024-04-18T21:52:51.332 INFO:tasks.rados.rados.0.smithi097.stdout:302: finishing write tid 1 to smithi09731216-13 2024-04-18T21:52:51.332 INFO:tasks.rados.rados.0.smithi097.stdout:302: finishing write tid 2 to smithi09731216-13 2024-04-18T21:52:51.333 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:51.322+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:51.333 INFO:tasks.rados.rados.0.smithi097.stdout:291: expect (ObjNum 113 snap 23 seq_num 113) 2024-04-18T21:52:51.409 INFO:tasks.rados.rados.0.smithi097.stdout:298: finishing write tid 3 to smithi09731216-35 2024-04-18T21:52:51.426 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:51.422+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:51.466 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:51.466 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:51.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.466 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:749 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~458752 in=458752b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:51.466 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.066099+0000", 2024-04-18T21:52:51.466 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.39743998400000002, 2024-04-18T21:52:51.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39749843600000001, 2024-04-18T21:52:51.466 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 749 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066099+0000", 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066099+0000", 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066099+0000", 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4299999999999998e-07 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066632+0000", 2024-04-18T21:52:51.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00053204999999999995 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066641+0000", 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3610000000000007e-06 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066647+0000", 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7869999999999998e-06 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.218362+0000", 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15171483999999999 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.218541+0000", 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000179848 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.220248+0000", 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00170695 2024-04-18T21:52:51.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.226744+0000", 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064952409999999997 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=150, last_complete=94'32, committed=1, applied=1))", 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.323896+0000", 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.139643191, 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13994726900000001, 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.471 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323896+0000", 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323896+0000", 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323902+0000", 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4330000000000001e-06 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323912+0000", 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7429999999999997e-06 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323939+0000", 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7389000000000002e-05 2024-04-18T21:52:51.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.424037+0000", 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100097357 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:750 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~81920 in=81920b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.066655+0000", 2024-04-18T21:52:51.474 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.39688440899999999, 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39732942999999998, 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 750 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066655+0000", 2024-04-18T21:52:51.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066655+0000", 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066655+0000", 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066709+0000", 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4398999999999999e-05 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066716+0000", 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7159999999999997e-06 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066720+0000", 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.754e-06 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.221359+0000", 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15463885899999999 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.221636+0000", 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027681199999999998 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.223309+0000", 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016729550000000001 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.227024+0000", 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003715005 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:751 3.2s0 3:44bf1f18:::smithi09731216-13:head [setxattr _header (57) in=64b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:51.479 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.066726+0000", 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.39681308700000001, 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39750049700000001, 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 751 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066726+0000", 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066726+0000", 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066726+0000", 2024-04-18T21:52:51.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2899999999999999e-07 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066729+0000", 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1830000000000001e-06 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066732+0000", 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.546e-06 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066735+0000", 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.869e-06 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.224227+0000", 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15749200199999999 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.224524+0000", 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000296569 2024-04-18T21:52:51.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.225960+0000", 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014364709999999999 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.232257+0000", 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006296683 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:752 3.2s0 3:44bf1f18:::smithi09731216-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.066739+0000", 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.39680016699999998, 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39773219700000001, 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for rw locks", 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 752 2024-04-18T21:52:51.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066739+0000", 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066739+0000", 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066739+0000", 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.84e-07 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066741+0000", 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.412e-06 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066746+0000", 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5160000000000002e-06 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066751+0000", 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6779999999999999e-06 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.226474+0000", 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.159723478 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.226628+0000", 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000154015 2024-04-18T21:52:51.488 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=149, last_complete=94'31, committed=1, applied=1))", 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.363606+0000", 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.099933234999999995, 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101060362, 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363606+0000", 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363606+0000", 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363606+0000", 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4600000000000005e-07 2024-04-18T21:52:51.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363611+0000", 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4170000000000002e-06 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363623+0000", 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.243e-05 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363632+0000", 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9029999999999993e-06 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=150, last_complete=94'32, committed=1, applied=1))", 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.363676+0000", 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.099862700999999998, 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101121298, 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.492 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363676+0000", 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363676+0000", 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363677+0000", 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4399999999999998e-07 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363682+0000", 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3359999999999997e-06 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363693+0000", 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1364e-05 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363719+0000", 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6384999999999999e-05 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=151, last_complete=94'33, committed=1, applied=1))", 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.363760+0000", 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.099779373000000005, 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10113593, 2024-04-18T21:52:51.495 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363760+0000", 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363760+0000", 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363760+0000", 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5700000000000002e-07 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363764+0000", 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9960000000000004e-06 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363776+0000", 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1326e-05 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363784+0000", 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1669999999999999e-06 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.498 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=148, last_complete=94'30, committed=1, applied=1))", 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.363810+0000", 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.099728541000000004, 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101157779, 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363810+0000", 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363810+0000", 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363811+0000", 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.13e-07 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363815+0000", 2024-04-18T21:52:51.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.557e-06 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363828+0000", 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3016e-05 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363835+0000", 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.627e-06 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.501 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=149, last_complete=94'31, committed=1, applied=1))", 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.363846+0000", 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.099693189000000001, 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10118922900000001, 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363846+0000", 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363846+0000", 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363846+0000", 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.53e-07 2024-04-18T21:52:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363849+0000", 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.117e-06 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363861+0000", 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1372999999999999e-05 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363868+0000", 2024-04-18T21:52:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.917e-06 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=150, last_complete=94'32, committed=1, applied=1))", 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.363877+0000", 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.099661979999999997, 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101225645, 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363877+0000", 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363877+0000", 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363877+0000", 2024-04-18T21:52:51.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6699999999999999e-07 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363881+0000", 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1700000000000001e-06 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363892+0000", 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0916e-05 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363899+0000", 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.34e-06 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=151, last_complete=94'33, committed=1, applied=1))", 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.363908+0000", 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.099631037000000006, 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101262353, 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.508 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363908+0000", 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363908+0000", 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363908+0000", 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.53e-07 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363912+0000", 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4000000000000001e-06 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363923+0000", 2024-04-18T21:52:51.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1015999999999999e-05 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.363929+0000", 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6170000000000003e-06 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=149, last_complete=94'31, committed=1, applied=1))", 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.374420+0000", 2024-04-18T21:52:51.511 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.089119335999999993, 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090819212999999996, 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374420+0000", 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374420+0000", 2024-04-18T21:52:51.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374421+0000", 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5000000000000001e-07 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374426+0000", 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4319999999999998e-06 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374441+0000", 2024-04-18T21:52:51.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5447e-05 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374453+0000", 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1212e-05 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=150, last_complete=94'32, committed=1, applied=1))", 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.374466+0000", 2024-04-18T21:52:51.514 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.089072625000000002, 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090839820000000002, 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374466+0000", 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374466+0000", 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374467+0000", 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.58e-07 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374470+0000", 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.315e-06 2024-04-18T21:52:51.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374480+0000", 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8509999999999994e-06 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374486+0000", 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.071e-06 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=151, last_complete=94'33, committed=1, applied=1))", 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.374497+0000", 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.089042354000000004, 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090876930999999994, 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374497+0000", 2024-04-18T21:52:51.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374497+0000", 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374497+0000", 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0199999999999998e-07 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374500+0000", 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7190000000000001e-06 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374509+0000", 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8789999999999992e-06 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374514+0000", 2024-04-18T21:52:51.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8150000000000002e-06 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=148, last_complete=94'30, committed=1, applied=1))", 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.374812+0000", 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.088727189999999997, 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090629845000000001, 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374812+0000", 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374812+0000", 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374813+0000", 2024-04-18T21:52:51.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0299999999999998e-07 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374818+0000", 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0259999999999998e-06 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374830+0000", 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2345e-05 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374838+0000", 2024-04-18T21:52:51.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5290000000000008e-06 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=149, last_complete=94'31, committed=1, applied=1))", 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.374940+0000", 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.088599444999999999, 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090574593999999994, 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374940+0000", 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374940+0000", 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374940+0000", 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4399999999999999e-07 2024-04-18T21:52:51.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374943+0000", 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.117e-06 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374953+0000", 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0009e-05 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374963+0000", 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3079999999999997e-06 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=150, last_complete=94'32, committed=1, applied=1))", 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.374976+0000", 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.088562752999999994, 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090611739999999996, 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.527 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374976+0000", 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374976+0000", 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374977+0000", 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.39e-07 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374979+0000", 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5890000000000001e-06 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374989+0000", 2024-04-18T21:52:51.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.38e-06 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374995+0000", 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2920000000000001e-06 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=151, last_complete=94'33, committed=1, applied=1))", 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.375021+0000", 2024-04-18T21:52:51.530 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.088518371999999998, 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090635144000000001, 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375021+0000", 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375021+0000", 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375021+0000", 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5100000000000001e-07 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375024+0000", 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.066e-06 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375033+0000", 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5280000000000003e-06 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375039+0000", 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5869999999999998e-06 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=149, last_complete=94'31, committed=1, applied=1))", 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.375057+0000", 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.088481533000000001, 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090669499000000001, 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375057+0000", 2024-04-18T21:52:51.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375057+0000", 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375058+0000", 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.27e-07 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375061+0000", 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8940000000000002e-06 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375071+0000", 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9240000000000002e-06 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375077+0000", 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1140000000000002e-06 2024-04-18T21:52:51.536 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=150, last_complete=94'32, committed=1, applied=1))", 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.375088+0000", 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.088451460999999995, 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090779159999999998, 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375088+0000", 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375088+0000", 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375088+0000", 2024-04-18T21:52:51.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6e-07 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375090+0000", 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.283e-06 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375099+0000", 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5290000000000008e-06 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375104+0000", 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4829999999999999e-06 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=151, last_complete=94'33, committed=1, applied=1))", 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.375158+0000", 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.088380790000000001, 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090887286999999997, 2024-04-18T21:52:51.540 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375158+0000", 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375158+0000", 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375159+0000", 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8800000000000003e-07 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375162+0000", 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9330000000000001e-06 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375173+0000", 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1080999999999999e-05 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.375179+0000", 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0870000000000003e-06 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=151, last_complete=94'33, committed=1, applied=1))", 2024-04-18T21:52:51.543 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.424083+0000", 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.039456074000000001, 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042038355999999999, 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.424083+0000", 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.424083+0000", 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.424084+0000", 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.147e-06 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.424094+0000", 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5529999999999998e-06 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.424120+0000", 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6699e-05 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:748 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~507904 in=507904b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.065761+0000", 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.39777845000000001, 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40043383599999999, 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.546 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 748 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.065761+0000", 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.065761+0000", 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.065761+0000", 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7899999999999999e-07 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066076+0000", 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031490799999999997 2024-04-18T21:52:51.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066085+0000", 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6459999999999999e-06 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066089+0000", 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6360000000000002e-06 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.112513+0000", 2024-04-18T21:52:51.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046423352000000001 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.112893+0000", 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00037991899999999999 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.115390+0000", 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024979429999999999 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.217476+0000", 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102085156 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=148, last_complete=94'30, committed=1, applied=1))", 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.274285+0000", 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.18925407799999999, 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19202603400000001, 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.551 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274285+0000", 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274285+0000", 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274286+0000", 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2699999999999998e-07 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274290+0000", 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6519999999999997e-06 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274304+0000", 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4049000000000001e-05 2024-04-18T21:52:51.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374388+0000", 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100083481 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 25 2024-04-18T21:52:51.554 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:51.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:307: expect (ObjNum 98 snap 19 seq_num 98) 2024-04-18T21:52:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:291: done (11 left) 2024-04-18T21:52:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:307: done (10 left) 2024-04-18T21:52:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:309: done (9 left) 2024-04-18T21:52:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 32 current snap is 24 2024-04-18T21:52:51.558 INFO:tasks.rados.rados.0.smithi097.stdout:310: seq_num 122 ranges {524288=507904,1032192=425984,1458176=442368,1900544=540672,2441216=147456} 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:310: writing smithi09731216-32 from 524288 to 1032192 tid 1 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:310: writing smithi09731216-32 from 1032192 to 1458176 tid 2 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:310: writing smithi09731216-32 from 1458176 to 1900544 tid 3 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:310: writing smithi09731216-32 from 1900544 to 2441216 tid 4 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:310: writing smithi09731216-32 from 2441216 to 2588672 tid 5 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 19 current snap is 24 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:311: seq_num 123 ranges {0=524288,524288=573440,1097728=688128,1785856=294912} 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:311: writing smithi09731216-19 from 0 to 524288 tid 1 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:311: writing smithi09731216-19 from 524288 to 1097728 tid 2 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:311: writing smithi09731216-19 from 1097728 to 1785856 tid 3 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:311: writing smithi09731216-19 from 1785856 to 2080768 tid 4 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:312: setattr oid 36 current snap is 24 2024-04-18T21:52:51.559 INFO:tasks.rados.rados.0.smithi097.stdout:313: snap_remove snap 23 2024-04-18T21:52:51.560 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:51.550+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:51.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-18T21:52:51.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:51.657 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:51.654+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:51.683 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:51.683 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:51.684 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:51.684 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:51.684 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:51.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:51.757 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:51.754+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:51.814 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:51.815 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:51.828 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:51.828 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:51.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:51.828 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:51.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.828 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:97 3.4s0 3:209f2930:::smithi09731216-33:head [append 0~540672,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.839945708000002, 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4026191889999999, 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.829 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985858+0000", 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.92e-06 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987540+0000", 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001681463 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987566+0000", 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6043000000000001e-05 2024-04-18T21:52:51.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987586+0000", 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9930000000000001e-05 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087950+0000", 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100363824 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:51.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087956+0000", 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9200000000000001e-06 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388374+0000", 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3004186340000001 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388475+0000", 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100455 2024-04-18T21:52:51.833 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:98 3.4s0 3:209f2930:::smithi09731216-33:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.838184306000002, 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.401002842, 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.834 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 98 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987618+0000", 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5e-06 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987625+0000", 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6769999999999998e-06 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987637+0000", 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1693e-05 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987646+0000", 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3700000000000001e-06 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088140+0000", 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10049390900000001 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088145+0000", 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3600000000000004e-06 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388530+0000", 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300384269 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388620+0000", 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0063999999999997e-05 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:99 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.838141566000004, 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503241456, 2024-04-18T21:52:51.839 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987661+0000", 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5999999999999991e-07 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987664+0000", 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6459999999999999e-06 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987675+0000", 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0967e-05 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987684+0000", 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6929999999999996e-06 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088241+0000", 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100557049 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088244+0000", 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2729999999999998e-06 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388746+0000", 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300502061 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388878+0000", 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131533 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490901+0000", 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102023274 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.844 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:117 3.bs0 3:d4cd3926:::smithi09731216-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.638072557000001, 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.520974363, 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.845 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 117 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187729+0000", 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187730+0000", 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.127e-06 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187735+0000", 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.536e-06 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187752+0000", 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6186999999999999e-05 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.187762+0000", 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0587e-05 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281120+0000", 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093357525999999996 2024-04-18T21:52:51.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.281296+0000", 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017659300000000001 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405205+0000", 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.123908465 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.405449+0000", 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000244505 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.708703+0000", 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30325383700000003 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.850 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:126 3.14s0 3:2c76b789:::smithi09731216-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 63.319544045000001, 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50118609800000002, 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 126 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506257+0000", 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506258+0000", 2024-04-18T21:52:51.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506267+0000", 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7159999999999992e-06 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506287+0000", 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9542999999999999e-05 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.506301+0000", 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4572999999999999e-05 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511483+0000", 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051813989999999997 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.511616+0000", 2024-04-18T21:52:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013376499999999999 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803750+0000", 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29213323699999999 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.803998+0000", 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024861799999999998 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.007443+0000", 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20344494699999999 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.16s4 32/31 ECSubWrite(tid=15, reqid=client.4509.0:110, at_version=32'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 62.737644930000002, 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.42111731499999999, 2024-04-18T21:52:51.856 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088156+0000", 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088157+0000", 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088207+0000", 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0584999999999998e-05 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088241+0000", 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3784e-05 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.088269+0000", 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8314999999999999e-05 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408130+0000", 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31986045299999999 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.408140+0000", 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0248000000000001e-05 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509225+0000", 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10108521500000001 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.509274+0000", 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8374999999999997e-05 2024-04-18T21:52:51.860 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:216 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.809888549999997, 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51394664400000001, 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.861 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015913+0000", 2024-04-18T21:52:51.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015916+0000", 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0130000000000002e-06 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015930+0000", 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4705000000000001e-05 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015964+0000", 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4072000000000002e-05 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.015996+0000", 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1198999999999997e-05 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016142+0000", 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014630900000000001 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016155+0000", 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2581e-05 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.020038+0000", 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038838789999999998 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.020191+0000", 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000152559 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.021481+0000", 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012896680000000001 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128155+0000", 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106673879 2024-04-18T21:52:51.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.529681+0000", 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40152656199999998 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.529859+0000", 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017821800000000001 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:217 3.bs0 3:d90895f4:::smithi09731216-9:head [rollback 2] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.809751515999999, 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.51416292299999999, 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016050+0000", 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016051+0000", 2024-04-18T21:52:51.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6470000000000001e-06 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016062+0000", 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0451999999999999e-05 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016085+0000", 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2776e-05 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016103+0000", 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8131e-05 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016278+0000", 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017520200000000001 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016289+0000", 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1048e-05 2024-04-18T21:52:51.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.023057+0000", 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067683750000000001 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.023334+0000", 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027651500000000002 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.024976+0000", 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016423379999999999 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128157+0000", 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10318049899999999 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.530073+0000", 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40191629299999998 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.530213+0000", 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013964700000000001 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:218 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.809668211000002, 2024-04-18T21:52:51.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61382267800000001, 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 218 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016134+0000", 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016141+0000", 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5719999999999999e-06 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016161+0000", 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9814e-05 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016177+0000", 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6028e-05 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016365+0000", 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018808400000000001 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016384+0000", 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-05 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026326+0000", 2024-04-18T21:52:51.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0099418909999999996 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026591+0000", 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026506899999999999 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.027603+0000", 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010121799999999999 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128223+0000", 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100619874 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629817+0000", 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50159395299999998 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629956+0000", 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013901300000000001 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.880 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:298 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~720896] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 53.764330667999999, 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.422019223, 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.881 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 298 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061471+0000", 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.061474+0000", 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2770000000000001e-06 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063120+0000", 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001645903 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063156+0000", 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.642e-05 2024-04-18T21:52:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063174+0000", 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7621e-05 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063316+0000", 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000141753 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.063330+0000", 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4324999999999999e-05 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.166460+0000", 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103130208 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.166636+0000", 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175792 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.170069+0000", 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003432778 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279315+0000", 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.109246564 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.483316+0000", 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20400066 2024-04-18T21:52:51.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.483490+0000", 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017392199999999999 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:299 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 53.761486486000003, 2024-04-18T21:52:51.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.518657543, 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 299 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:51.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064315+0000", 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.064317+0000", 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5900000000000002e-06 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065148+0000", 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000830709 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065184+0000", 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5738999999999998e-05 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065209+0000", 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5627e-05 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065508+0000", 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029824799999999998 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065539+0000", 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0871999999999999e-05 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.172826+0000", 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.107287676 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.174217+0000", 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001390891 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.177064+0000", 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028466120000000001 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279319+0000", 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102254998 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.582844+0000", 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30352516899999998 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.582972+0000", 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000128412 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:300 3.bs0 3:d90895f4:::smithi09731216-9:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 53.760541893000003, 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.71857623199999998, 2024-04-18T21:52:51.894 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.895 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.895 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.897 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.897 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.897 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.897 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 300 2024-04-18T21:52:51.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.897 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065261+0000", 2024-04-18T21:52:51.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9690000000000001e-06 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065272+0000", 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0545e-05 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065299+0000", 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6927000000000002e-05 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065319+0000", 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0035e-05 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165723+0000", 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100404099 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165747+0000", 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.438e-05 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279995+0000", 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.114247892 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280276+0000", 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028071299999999998 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281155+0000", 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087942000000000001 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.381976+0000", 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100820639 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783683+0000", 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40170656199999999 2024-04-18T21:52:51.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783836+0000", 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000153051 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:301 3.bs0 3:d90895f4:::smithi09731216-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 53.760409738999996, 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.82307189599999997, 2024-04-18T21:52:51.903 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 301 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065393+0000", 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.553e-06 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065401+0000", 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9719999999999992e-06 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065425+0000", 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3547e-05 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065447+0000", 2024-04-18T21:52:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2234000000000001e-05 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165843+0000", 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100395994 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165860+0000", 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6871000000000002e-05 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281656+0000", 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11579645099999999 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281793+0000", 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013725 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783958+0000", 2024-04-18T21:52:51.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50216431100000003 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.784112+0000", 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015411799999999999 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.888463+0000", 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10435159500000001 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:422 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~294912] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.378921286999997, 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61132010699999995, 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.910 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 422 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446882+0000", 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5459999999999998e-06 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447318+0000", 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00043564599999999998 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447349+0000", 2024-04-18T21:52:51.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0888e-05 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447365+0000", 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5593000000000001e-05 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447499+0000", 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013438999999999999 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447509+0000", 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6509999999999996e-06 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.556315+0000", 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10880680299999999 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.556603+0000", 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028725400000000002 2024-04-18T21:52:51.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.558288+0000", 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001684786 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638308+0000", 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.080020547999999997 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058105+0000", 2024-04-18T21:52:51.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41979719399999998 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058200+0000", 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4808000000000004e-05 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:423 3.2s0 3:44bf1f18:::smithi09731216-13:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.378404144000001, 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61117605200000003, 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 423 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447398+0000", 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.353e-06 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447416+0000", 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7353000000000001e-05 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447431+0000", 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5645e-05 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447441+0000", 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2690000000000005e-06 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447569+0000", 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012837999999999999 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447573+0000", 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3270000000000002e-06 2024-04-18T21:52:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.559335+0000", 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11176207000000001 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.559597+0000", 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026142 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.560692+0000", 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010952799999999999 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638310+0000", 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.077617829999999999 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058483+0000", 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.42017316199999999 2024-04-18T21:52:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058573+0000", 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9963000000000006e-05 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:424 3.2s0 3:44bf1f18:::smithi09731216-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.378306051000003, 2024-04-18T21:52:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.81405277300000001, 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 424 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447496+0000", 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1699999999999997e-07 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447501+0000", 2024-04-18T21:52:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5059999999999997e-06 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447513+0000", 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2592000000000001e-05 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447526+0000", 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2309e-05 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.547672+0000", 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10014685800000001 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.547684+0000", 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1232e-05 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.756870+0000", 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.209186237 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.756965+0000", 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5280999999999998e-05 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.059024+0000", 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30205871899999998 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.059128+0000", 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010387600000000001 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.261548+0000", 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.202420246 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:434 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:51.929 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.378257675, 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.58875951599999998, 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 434 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447544+0000", 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999998e-07 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447549+0000", 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7740000000000004e-06 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447560+0000", 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0522e-05 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447569+0000", 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.312e-06 2024-04-18T21:52:51.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447614+0000", 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5370000000000001e-05 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447620+0000", 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.502e-06 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638433+0000", 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.190813609 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638582+0000", 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014889800000000001 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.036303+0000", 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39772072600000002 2024-04-18T21:52:51.934 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:435 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.378211252, 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.58911970899999999, 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.935 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 435 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447591+0000", 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2270000000000001e-06 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447598+0000", 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6630000000000004e-06 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447616+0000", 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8587999999999999e-05 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447631+0000", 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4326e-05 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447673+0000", 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2159000000000001e-05 2024-04-18T21:52:51.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447677+0000", 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0799999999999999e-06 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639051+0000", 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19137447299999999 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639173+0000", 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121443 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.036710+0000", 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39753674999999999 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:436 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:51.940 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.378149010000001, 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.59017562599999995, 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 436 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447653+0000", 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1540000000000001e-06 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447661+0000", 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7449999999999995e-06 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447679+0000", 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8263000000000001e-05 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447693+0000", 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3642000000000001e-05 2024-04-18T21:52:51.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447729+0000", 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6124000000000002e-05 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447733+0000", 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9620000000000004e-06 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639451+0000", 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19171812099999999 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639522+0000", 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0956000000000003e-05 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.037828+0000", 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39830565899999998 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.945 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=148, last_complete=94'30, committed=1, applied=1))", 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.274285+0000", 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.55151627999999997, 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.44937308500000001, 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274285+0000", 2024-04-18T21:52:51.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274285+0000", 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274286+0000", 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2699999999999998e-07 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274290+0000", 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6519999999999997e-06 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.274304+0000", 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4049000000000001e-05 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.374388+0000", 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100083481 2024-04-18T21:52:51.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.723636+0000", 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.34924782700000001 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.723658+0000", 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2249e-05 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.949 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:51.950 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:51.950 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:51.951 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:51.951 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:51.952 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:51.952 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:51.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:51.953 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:51.953 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:51.954 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:51.946+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:52.052 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:52.046+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:52.076 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:52.076 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.14s3 95/94 ECSubWrite(tid=319, reqid=osd.4.0:319, at_version=95'14, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:52.081611+0000", 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0023128720000000001, 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023530780000000002, 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:52.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.081611+0000", 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.081611+0000", 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.081612+0000", 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.637e-06 2024-04-18T21:52:52.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.081649+0000", 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6689000000000001e-05 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.081678+0000", 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9036000000000001e-05 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.081693+0000", 2024-04-18T21:52:52.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4993e-05 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.5s5 95/90 ECSubRead(tid=202, to_read={3:a2c4e662:::smithi09731216-50:head=0,262144,0}, subchunks={3:a2c4e662:::smithi09731216-50:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:52.047332+0000", 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.036591564, 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.03673183, 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:52.086 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047332+0000", 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047332+0000", 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047333+0000", 2024-04-18T21:52:52.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047345+0000", 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2734e-05 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047367+0000", 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1138999999999999e-05 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:52.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047381+0000", 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4596000000000001e-05 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "pg_backfill(progress 3.4s7 e 95/95 lb 3:209f2930:::smithi09731216-33:18)", 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:52.047618+0000", 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.036305667999999999, 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.036521314999999999, 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:52.089 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047618+0000", 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047618+0000", 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047618+0000", 2024-04-18T21:52:52.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0499999999999999e-07 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047631+0000", 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2918000000000001e-05 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.047641+0000", 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5249999999999994e-06 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:52.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:52.092 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:52.092 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-04-18T21:52:52.092 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:52.097 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:52.153 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:52.151+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:52.174 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:52.175 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:52.204 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:52.204 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:52.219 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:52.219 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:52.219 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:52.220 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:52.220 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:52.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:52.235 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:52.236 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:52.264 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:52.264 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:52.277 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:52.275+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:52.291 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:52.292 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:52.319 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:52.319 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:52.344 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:52.344 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:52.354 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:52.351+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:52.362 INFO:teuthology.orchestra.run.smithi097.stderr:set 3.2 pg_upmap mapping to [2,6,0,9,11,8,1,4] 2024-04-18T21:52:52.363 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is set 2024-04-18T21:52:52.369 INFO:tasks.rados.rados.0.smithi097.stdout:313: done (12 left) 2024-04-18T21:52:52.369 INFO:tasks.rados.rados.0.smithi097.stdout:314: copy_from oid 12 from oid 23 current snap is 24 2024-04-18T21:52:52.369 INFO:tasks.rados.rados.0.smithi097.stdout:298: finishing write tid 4 to smithi09731216-35 2024-04-18T21:52:52.369 INFO:tasks.rados.rados.0.smithi097.stdout:298: finishing write tid 5 to smithi09731216-35 2024-04-18T21:52:52.369 INFO:tasks.rados.rados.0.smithi097.stdout:306: finishing write tid 1 to smithi09731216-49 2024-04-18T21:52:52.369 INFO:tasks.rados.rados.0.smithi097.stdout:306: finishing write tid 2 to smithi09731216-49 2024-04-18T21:52:52.369 INFO:tasks.rados.rados.0.smithi097.stdout:301: expect (ObjNum 114 snap 23 seq_num 114) 2024-04-18T21:52:52.408 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:52.408 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:52.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:52.456 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:52.450+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:52.525 INFO:tasks.rados.rados.0.smithi097.stdout:294: finishing rollback tid 0 to smithi09731216-20 2024-04-18T21:52:52.525 INFO:tasks.rados.rados.0.smithi097.stdout:304: expect (ObjNum 84 snap 14 seq_num 84) 2024-04-18T21:52:52.556 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:52.556 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:52.556 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:52.550+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:52.562 INFO:tasks.rados.rados.0.smithi097.stdout:303: finishing write tid 2 to smithi09731216-11 2024-04-18T21:52:52.562 INFO:tasks.rados.rados.0.smithi097.stdout:303: finishing write tid 3 to smithi09731216-11 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:303: finishing write tid 4 to smithi09731216-11 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:302: finishing write tid 3 to smithi09731216-13 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:302: finishing write tid 4 to smithi09731216-13 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:303: finishing write tid 5 to smithi09731216-11 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:294: finishing rollback tid 1 to smithi09731216-20 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:294: finishing rollback tid 2 to smithi09731216-20 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 20 v 155 (ObjNum 107 snap 19 seq_num 107) dirty exists 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:308: finishing rollback tid 1 to smithi09731216-3 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 3 v 122 (ObjNum 161 snap 0 seq_num 1) dirty dne 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:302: finishing write tid 5 to smithi09731216-13 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:302: finishing write tid 6 to smithi09731216-13 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:298: finishing write tid 6 to smithi09731216-35 2024-04-18T21:52:52.563 INFO:tasks.rados.rados.0.smithi097.stdout:298: oid 35 updating version 0 to 33 2024-04-18T21:52:52.564 INFO:tasks.rados.rados.0.smithi097.stdout:298: oid 35 updating version 33 to 34 2024-04-18T21:52:52.564 INFO:tasks.rados.rados.0.smithi097.stdout:298: oid 35 version 34 is already newer than 32 2024-04-18T21:52:52.564 INFO:tasks.rados.rados.0.smithi097.stdout:298: oid 35 updating version 34 to 36 2024-04-18T21:52:52.564 INFO:tasks.rados.rados.0.smithi097.stdout:298: oid 35 version 36 is already newer than 35 2024-04-18T21:52:52.564 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 35 v 36 (ObjNum 118 snap 23 seq_num 118) dirty exists 2024-04-18T21:52:52.564 INFO:tasks.rados.rados.0.smithi097.stdout:298: left oid 35 (ObjNum 118 snap 23 seq_num 118) 2024-04-18T21:52:52.564 INFO:tasks.rados.rados.0.smithi097.stdout:310: finishing write tid 1 to smithi09731216-32 2024-04-18T21:52:52.564 INFO:tasks.rados.rados.0.smithi097.stdout:310: finishing write tid 2 to smithi09731216-32 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 36 v 19 (ObjNum 273 snap 0 seq_num 0) dirty exists 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:303: finishing write tid 6 to smithi09731216-11 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:303: oid 11 updating version 0 to 122 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:303: oid 11 version 122 is already newer than 121 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:303: oid 11 updating version 122 to 123 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:303: oid 11 updating version 123 to 124 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:303: oid 11 version 124 is already newer than 120 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 11 v 124 (ObjNum 120 snap 23 seq_num 120) dirty exists 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:303: left oid 11 (ObjNum 120 snap 23 seq_num 120) 2024-04-18T21:52:52.565 INFO:tasks.rados.rados.0.smithi097.stdout:300: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-18T21:52:52.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:52.583 INFO:tasks.rados.rados.0.smithi097.stdout:310: finishing write tid 3 to smithi09731216-32 2024-04-18T21:52:52.583 INFO:tasks.rados.rados.0.smithi097.stdout:302: finishing write tid 7 to smithi09731216-13 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:302: oid 13 updating version 0 to 32 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:302: oid 13 version 32 is already newer than 31 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:302: oid 13 version 32 is already newer than 30 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:302: oid 13 updating version 32 to 33 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:302: oid 13 version 33 is already newer than 29 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:302: oid 13 version 33 is already newer than 28 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 13 v 33 (ObjNum 119 snap 23 seq_num 119) dirty exists 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:302: left oid 13 (ObjNum 119 snap 23 seq_num 119) 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:306: finishing write tid 3 to smithi09731216-49 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:306: finishing write tid 4 to smithi09731216-49 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:306: finishing write tid 5 to smithi09731216-49 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:310: finishing write tid 4 to smithi09731216-32 2024-04-18T21:52:52.584 INFO:tasks.rados.rados.0.smithi097.stdout:310: finishing write tid 5 to smithi09731216-32 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:294: done (12 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:298: done (11 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:300: done (10 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:301: done (9 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:302: done (8 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:303: done (7 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:304: done (6 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:308: done (5 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:312: done (4 left) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:315: read oid 37 snap -1 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:315: expect (ObjNum 91 snap 18 seq_num 91) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:316: read oid 16 snap -1 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:316: expect (ObjNum 103 snap 19 seq_num 103) 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:317: read oid 1 snap -1 2024-04-18T21:52:52.585 INFO:tasks.rados.rados.0.smithi097.stdout:317: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:318: read oid 38 snap -1 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:318: expect (ObjNum 69 snap 4 seq_num 69) 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:319: delete oid 44 current snap is 24 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:306: finishing write tid 6 to smithi09731216-49 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:306: oid 49 updating version 0 to 26 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:306: oid 49 version 26 is already newer than 24 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:306: oid 49 version 26 is already newer than 22 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:306: oid 49 version 26 is already newer than 23 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:306: oid 49 version 26 is already newer than 25 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 49 v 26 (ObjNum 121 snap 23 seq_num 121) dirty exists 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:306: left oid 49 (ObjNum 121 snap 23 seq_num 121) 2024-04-18T21:52:52.586 INFO:tasks.rados.rados.0.smithi097.stdout:310: finishing write tid 6 to smithi09731216-32 2024-04-18T21:52:52.658 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:52.654+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:52.707 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:52.707 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:52.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:52.765 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:52.762+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:52.849 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:52.850 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:52.854 INFO:tasks.rados.rados.0.smithi097.stdout:318: expect (ObjNum 69 snap 4 seq_num 69) 2024-04-18T21:52:52.860 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:52.855+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:52.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:52.869 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:52.866+0000 7f8622e9b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:52.961 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:52.958+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:52.966 INFO:tasks.rados.rados.0.smithi097.stdout:317: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:52:52.967 INFO:tasks.rados.rados.0.smithi097.stdout:306: done (8 left) 2024-04-18T21:52:52.967 INFO:tasks.rados.rados.0.smithi097.stdout:317: done (7 left) 2024-04-18T21:52:52.967 INFO:tasks.rados.rados.0.smithi097.stdout:318: done (6 left) 2024-04-18T21:52:52.967 INFO:tasks.rados.rados.0.smithi097.stdout:319: done (5 left) 2024-04-18T21:52:52.967 INFO:tasks.rados.rados.0.smithi097.stdout:320: delete oid 42 current snap is 24 2024-04-18T21:52:52.987 INFO:tasks.rados.rados.0.smithi097.stdout:314: finishing copy_from to smithi09731216-12 2024-04-18T21:52:52.987 INFO:tasks.rados.rados.0.smithi097.stdout:314: got expected ENOENT (src dne) 2024-04-18T21:52:52.987 INFO:tasks.rados.rados.0.smithi097.stdout:314: finishing copy_from racing read to smithi09731216-12 2024-04-18T21:52:52.989 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:52.989 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:53.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:53.062 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:53.059+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:53.072 INFO:tasks.rados.rados.0.smithi097.stdout:310: finishing write tid 7 to smithi09731216-32 2024-04-18T21:52:53.072 INFO:tasks.rados.rados.0.smithi097.stdout:310: oid 32 updating version 0 to 50 2024-04-18T21:52:53.072 INFO:tasks.rados.rados.0.smithi097.stdout:310: oid 32 updating version 50 to 51 2024-04-18T21:52:53.072 INFO:tasks.rados.rados.0.smithi097.stdout:310: oid 32 version 51 is already newer than 49 2024-04-18T21:52:53.072 INFO:tasks.rados.rados.0.smithi097.stdout:310: oid 32 updating version 51 to 53 2024-04-18T21:52:53.072 INFO:tasks.rados.rados.0.smithi097.stdout:310: oid 32 version 53 is already newer than 48 2024-04-18T21:52:53.072 INFO:tasks.rados.rados.0.smithi097.stdout:310: oid 32 version 53 is already newer than 52 2024-04-18T21:52:53.073 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 32 v 53 (ObjNum 122 snap 24 seq_num 122) dirty exists 2024-04-18T21:52:53.073 INFO:tasks.rados.rados.0.smithi097.stdout:310: left oid 32 (ObjNum 122 snap 24 seq_num 122) 2024-04-18T21:52:53.107 INFO:tasks.rados.rados.0.smithi097.stdout:311: finishing write tid 1 to smithi09731216-19 2024-04-18T21:52:53.152 INFO:tasks.rados.rados.0.smithi097.stdout:315: expect (ObjNum 91 snap 18 seq_num 91) 2024-04-18T21:52:53.163 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:53.159+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:53.168 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:53.168 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:53.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:53.219 INFO:tasks.rados.rados.0.smithi097.stdout:310: done (5 left) 2024-04-18T21:52:53.219 INFO:tasks.rados.rados.0.smithi097.stdout:314: done (4 left) 2024-04-18T21:52:53.219 INFO:tasks.rados.rados.0.smithi097.stdout:315: done (3 left) 2024-04-18T21:52:53.219 INFO:tasks.rados.rados.0.smithi097.stdout:320: done (2 left) 2024-04-18T21:52:53.219 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 29 current snap is 24 2024-04-18T21:52:53.219 INFO:tasks.rados.rados.0.smithi097.stdout:321: seq_num 126 ranges {0=573440,573440=622592,1196032=622592,1818624=311296} 2024-04-18T21:52:53.228 INFO:tasks.rados.rados.0.smithi097.stdout:321: writing smithi09731216-29 from 0 to 573440 tid 1 2024-04-18T21:52:53.238 INFO:tasks.rados.rados.0.smithi097.stdout:321: writing smithi09731216-29 from 573440 to 1196032 tid 2 2024-04-18T21:52:53.249 INFO:tasks.rados.rados.0.smithi097.stdout:321: writing smithi09731216-29 from 1196032 to 1818624 tid 3 2024-04-18T21:52:53.255 INFO:tasks.rados.rados.0.smithi097.stdout:321: writing smithi09731216-29 from 1818624 to 2129920 tid 4 2024-04-18T21:52:53.255 INFO:tasks.rados.rados.0.smithi097.stdout:322: delete oid 40 current snap is 24 2024-04-18T21:52:53.264 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:53.259+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:53.339 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:53.339 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:53.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:53.364 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:53.363+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:53.413 INFO:tasks.rados.rados.0.smithi097.stdout:311: finishing write tid 2 to smithi09731216-19 2024-04-18T21:52:53.413 INFO:tasks.rados.rados.0.smithi097.stdout:311: finishing write tid 3 to smithi09731216-19 2024-04-18T21:52:53.466 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:53.463+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:53.484 INFO:tasks.rados.rados.0.smithi097.stdout:322: done (3 left) 2024-04-18T21:52:53.484 INFO:tasks.rados.rados.0.smithi097.stdout:323: snap_remove snap 21 2024-04-18T21:52:53.526 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:53.527 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:53.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:53.585 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:53.582+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:53.667 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:53.663+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:53.678 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:53.674+0000 7f8622e9b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,5 to read 0,1,2,3,4,5,6,7 2024-04-18T21:52:53.711 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:53.711 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:53.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:53.768 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:53.763+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:53.849 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:53.849 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:53.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:52:53.869 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:53.866+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:53.970 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:53.966+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:53.986 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:53.986 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:54.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:52:54.090 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:54.086+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:54.169 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:54.169 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:54.171 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:54.166+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:54.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:52:54.272 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:54.266+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:54.353 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:54.353 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:54.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:52:54.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:52:54.370 INFO:tasks.rados.rados.0.smithi097.stdout:323: done (3 left) 2024-04-18T21:52:54.370 INFO:tasks.rados.rados.0.smithi097.stdout:324: delete oid 23 current snap is 24 2024-04-18T21:52:54.370 INFO:tasks.rados.rados.0.smithi097.stdout:311: finishing write tid 4 to smithi09731216-19 2024-04-18T21:52:54.370 INFO:tasks.rados.rados.0.smithi097.stdout:311: finishing write tid 5 to smithi09731216-19 2024-04-18T21:52:54.370 INFO:tasks.rados.rados.0.smithi097.stdout:311: finishing write tid 6 to smithi09731216-19 2024-04-18T21:52:54.370 INFO:tasks.rados.rados.0.smithi097.stdout:311: oid 19 updating version 0 to 59 2024-04-18T21:52:54.370 INFO:tasks.rados.rados.0.smithi097.stdout:311: oid 19 updating version 59 to 60 2024-04-18T21:52:54.371 INFO:tasks.rados.rados.0.smithi097.stdout:311: oid 19 version 60 is already newer than 57 2024-04-18T21:52:54.371 INFO:tasks.rados.rados.0.smithi097.stdout:311: oid 19 updating version 60 to 61 2024-04-18T21:52:54.371 INFO:tasks.rados.rados.0.smithi097.stdout:311: oid 19 version 61 is already newer than 58 2024-04-18T21:52:54.371 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 19 v 61 (ObjNum 123 snap 24 seq_num 123) dirty exists 2024-04-18T21:52:54.371 INFO:tasks.rados.rados.0.smithi097.stdout:311: left oid 19 (ObjNum 123 snap 24 seq_num 123) 2024-04-18T21:52:54.371 INFO:tasks.rados.rados.0.smithi097.stdout:321: finishing write tid 1 to smithi09731216-29 2024-04-18T21:52:54.371 INFO:tasks.rados.rados.0.smithi097.stdout:321: finishing write tid 2 to smithi09731216-29 2024-04-18T21:52:54.371 INFO:tasks.rados.rados.0.smithi097.stdout:321: finishing write tid 3 to smithi09731216-29 2024-04-18T21:52:54.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-18T21:52:54.413 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:54.410+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:54.473 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:54.471+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:54.505 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:54.505 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:54.518 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:54.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:52:54.535 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:54.535 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:54.575 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:54.571+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:54.601 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:54.601 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:54.675 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:54.670+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:54.687 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:54.687 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:54.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:52:54.738 INFO:tasks.rados.rados.0.smithi097.stdout:321: finishing write tid 4 to smithi09731216-29 2024-04-18T21:52:54.738 INFO:tasks.rados.rados.0.smithi097.stdout:321: finishing write tid 5 to smithi09731216-29 2024-04-18T21:52:54.776 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:54.774+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:54.827 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:54.828 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:54.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:52:54.877 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:54.874+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:54.879 INFO:tasks.rados.rados.0.smithi097.stdout:321: finishing write tid 6 to smithi09731216-29 2024-04-18T21:52:54.879 INFO:tasks.rados.rados.0.smithi097.stdout:321: oid 29 updating version 0 to 181 2024-04-18T21:52:54.879 INFO:tasks.rados.rados.0.smithi097.stdout:321: oid 29 version 181 is already newer than 178 2024-04-18T21:52:54.879 INFO:tasks.rados.rados.0.smithi097.stdout:321: oid 29 version 181 is already newer than 179 2024-04-18T21:52:54.879 INFO:tasks.rados.rados.0.smithi097.stdout:321: oid 29 version 181 is already newer than 180 2024-04-18T21:52:54.879 INFO:tasks.rados.rados.0.smithi097.stdout:321: oid 29 version 181 is already newer than 177 2024-04-18T21:52:54.880 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 29 v 181 (ObjNum 126 snap 24 seq_num 126) dirty exists 2024-04-18T21:52:54.880 INFO:tasks.rados.rados.0.smithi097.stdout:321: left oid 29 (ObjNum 126 snap 24 seq_num 126) 2024-04-18T21:52:54.924 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:54.924 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:54.947 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:54.948 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:54.978 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:54.974+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:54.979 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:54.979 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:55.011 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:55.011 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:55.013 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:55.013 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:55.013 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:55.013 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:55.027 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:52:55.038 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:55.039 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:55.048 INFO:tasks.rados.rados.0.smithi097.stdout:311: done (3 left) 2024-04-18T21:52:55.048 INFO:tasks.rados.rados.0.smithi097.stdout:321: done (2 left) 2024-04-18T21:52:55.048 INFO:tasks.rados.rados.0.smithi097.stdout:324: done (1 left) 2024-04-18T21:52:55.048 INFO:tasks.rados.rados.0.smithi097.stdout:325: copy_from oid 5 from oid 27 current snap is 24 2024-04-18T21:52:55.048 INFO:tasks.rados.rados.0.smithi097.stdout:326: rollback oid 18 current snap is 24 2024-04-18T21:52:55.048 INFO:tasks.rados.rados.0.smithi097.stdout:326: done (2 left) 2024-04-18T21:52:55.048 INFO:tasks.rados.rados.0.smithi097.stdout:327: delete oid 37 current snap is 24 2024-04-18T21:52:55.068 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:55.068 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:55.080 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:55.075+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:55.094 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:55.103 INFO:tasks.rados.rados.0.smithi097.stdout:316: expect (ObjNum 103 snap 19 seq_num 103) 2024-04-18T21:52:55.121 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:55.121 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:55.143 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:55.143 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:55.143 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:55.143 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:55.144 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:55.158 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:52:55.177 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:55.177 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:55.181 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:55.178+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:55.282 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:55.278+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:55.284 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:100 3.cs0 3:348e44d8:::smithi09731216-34:head [append 0~557056,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.621716825999997, 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50652407300000002, 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:52:55.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659953+0000", 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.821e-06 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660381+0000", 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042725800000000002 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660395+0000", 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.394e-05 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660404+0000", 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6520000000000001e-06 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760564+0000", 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10015995900000001 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760699+0000", 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013526 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762105+0000", 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00140595 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965039+0000", 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20293309400000001 2024-04-18T21:52:55.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166421+0000", 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201382373 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166476+0000", 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4765999999999997e-05 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:101 3.cs0 3:348e44d8:::smithi09731216-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.621247085999997, 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60956795200000002, 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660422+0000", 2024-04-18T21:52:55.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-07 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660426+0000", 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3830000000000002e-06 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660436+0000", 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8679999999999994e-06 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660443+0000", 2024-04-18T21:52:55.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2999999999999998e-06 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762872+0000", 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1024294 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762973+0000", 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001015 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763506+0000", 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053209199999999996 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965047+0000", 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20154100999999999 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269839+0000", 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.304792377 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269989+0000", 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015033399999999999 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:102 3.cs0 3:348e44d8:::smithi09731216-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.621197537, 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.92759091000000005, 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.296 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660472+0000", 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660476+0000", 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0879999999999997e-06 2024-04-18T21:52:55.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660493+0000", 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6963e-05 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660502+0000", 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9579999999999996e-06 2024-04-18T21:52:55.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763793+0000", 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10329121600000001 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763891+0000", 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8127000000000004e-05 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372039+0000", 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60814838199999999 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372189+0000", 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014991200000000001 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.588062+0000", 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.215872654 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:78 3.15s0 3:ab33d1b5:::smithi09731216-26:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.300652112999998, 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1076682130000002, 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78 2024-04-18T21:52:55.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981019+0000", 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.447e-06 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981025+0000", 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6810000000000001e-06 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981044+0000", 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8856999999999999e-05 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981056+0000", 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1921e-05 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164029+0000", 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18297308600000001 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164033+0000", 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0369999999999996e-06 2024-04-18T21:52:55.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885490+0000", 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7214569900000001 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885771+0000", 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028064300000000002 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.088685+0000", 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.202913551 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 40/26 ECSubWriteReply(tid=53, last_complete=40'21, committed=1, applied=1))", 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 57.760043553999999, 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.409140051, 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:55.307 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521625+0000", 2024-04-18T21:52:55.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0900000000000002e-07 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521629+0000", 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0169999999999998e-06 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521640+0000", 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0246e-05 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.521647+0000", 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9070000000000001e-06 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.929171+0000", 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40752403300000001 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:57.930765+0000", 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001594339 2024-04-18T21:52:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:463 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~638976] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.907579263999999, 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50349991299999997, 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 463 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374091+0000", 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.84e-06 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375137+0000", 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010459 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375156+0000", 2024-04-18T21:52:55.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978e-05 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375169+0000", 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2660000000000001e-05 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571128+0000", 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19595929000000001 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571311+0000", 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018294599999999999 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572699+0000", 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001388715 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675602+0000", 2024-04-18T21:52:55.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102902721 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877476+0000", 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20187386299999999 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877589+0000", 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.317 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:464 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~131072] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:55.317 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:55.317 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.906465679, 2024-04-18T21:52:55.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50427034999999998, 2024-04-18T21:52:55.317 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.317 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.317 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 464 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375204+0000", 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3650000000000001e-06 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375344+0000", 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013967800000000001 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375357+0000", 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3580999999999999e-05 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375367+0000", 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0117e-05 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573497+0000", 2024-04-18T21:52:55.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198129889 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573606+0000", 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010875 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574480+0000", 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00087374900000000005 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675605+0000", 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10112497400000001 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879333+0000", 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20372839500000001 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879473+0000", 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013985199999999999 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:465 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.906283144, 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60269858399999998, 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.323 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 465 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375386+0000", 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6300000000000004e-07 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375391+0000", 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.729e-06 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375401+0000", 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9569999999999997e-06 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375409+0000", 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6620000000000005e-06 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777143+0000", 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40173476200000002 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777239+0000", 2024-04-18T21:52:55.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5712999999999997e-05 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777695+0000", 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045595 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.778376+0000", 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068123800000000003 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.977952+0000", 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.199575432 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.978084+0000", 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000132278 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:466 3.cs0 3:3c73a4f8:::smithi09731216-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:55.328 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.906248456999997, 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006726617, 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 466 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375421+0000", 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.009e-06 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375424+0000", 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.495e-06 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375435+0000", 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0360000000000001e-05 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375442+0000", 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6389999999999992e-06 2024-04-18T21:52:55.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777929+0000", 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40248657999999998 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777993+0000", 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078687+0000", 2024-04-18T21:52:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30069407399999998 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078849+0000", 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162065 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.382147+0000", 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30329795300000001 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:696 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.8040269319999993, 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8198962060000001, 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 696 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.334 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:52:55.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8899999999999998e-07 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477645+0000", 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2030000000000001e-06 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477655+0000", 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8050000000000001e-06 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477663+0000", 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2459999999999997e-06 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.554664+0000", 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.077001179000000003 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.554666+0000", 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1399999999999998e-06 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.588198+0000", 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0335324410000002 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.588403+0000", 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020478799999999999 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.992774+0000", 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.404371174 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.993448+0000", 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067422999999999995 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.297538+0000", 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30408921100000003 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:733 3.cs0 3:348e44d8:::smithi09731216-34:head [stat,delete] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:50.487164+0000", 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.7945044750000001, 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40997382999999998, 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 733 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.487164+0000", 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.487164+0000", 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.487167+0000", 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0730000000000001e-06 2024-04-18T21:52:55.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.487180+0000", 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2612999999999999e-05 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.487210+0000", 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0173999999999999e-05 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.487226+0000", 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6025999999999999e-05 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.487393+0000", 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016743199999999999 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.487419+0000", 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6176000000000001e-05 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.500836+0000", 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013416587000000001 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.501131+0000", 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029479400000000002 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.503537+0000", 2024-04-18T21:52:55.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024062070000000001 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.506628+0000", 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030914129999999999 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.896882+0000", 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39025327700000001 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.897138+0000", 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025605799999999998 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:743 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~3768320,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:50.958326+0000", 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.3233428429999998, 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41477493199999999, 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.346 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 743 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958326+0000", 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958326+0000", 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958329+0000", 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0199999999999999e-06 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958342+0000", 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2943e-05 2024-04-18T21:52:55.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958373+0000", 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1449999999999999e-05 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958388+0000", 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5367e-05 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.959849+0000", 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014602319999999999 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.960416+0000", 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056786399999999996 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.373101+0000", 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41268405600000002 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:744 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~3768320,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.3232480769999997, 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52158899199999997, 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.351 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 744 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958421+0000", 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0499999999999999e-06 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958427+0000", 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0059999999999998e-06 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958440+0000", 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2269999999999999e-05 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958453+0000", 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2893e-05 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.961672+0000", 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032194329999999998 2024-04-18T21:52:55.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.962277+0000", 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060548399999999999 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.480009+0000", 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51773185600000005 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.355 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:745 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~3768320,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.3231761410000002, 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62853464000000003, 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 745 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958493+0000", 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7700000000000003e-07 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958498+0000", 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.096e-06 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958513+0000", 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4904e-05 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958522+0000", 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5760000000000006e-06 2024-04-18T21:52:55.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.064677+0000", 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106155732 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.065006+0000", 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032830000000000001 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.587027+0000", 2024-04-18T21:52:55.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52202115500000001 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:759 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2381927590000004, 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62612275900000003, 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.360 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 759 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043479+0000", 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8430000000000001e-06 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043492+0000", 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.322e-05 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043518+0000", 2024-04-18T21:52:55.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6567000000000002e-05 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043533+0000", 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4863e-05 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.201592+0000", 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.158059226 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.201907+0000", 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031459700000000001 2024-04-18T21:52:55.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.669598+0000", 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46769144299999998 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:760 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.238101984, 2024-04-18T21:52:55.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62795852399999996, 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 760 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043567+0000", 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6700000000000002e-07 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043573+0000", 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3499999999999996e-06 2024-04-18T21:52:55.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043585+0000", 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.205e-05 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043594+0000", 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8300000000000002e-06 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.202744+0000", 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15915027200000001 2024-04-18T21:52:55.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.202986+0000", 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000241883 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.671525+0000", 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.468539172 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:761 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2380620100000002, 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.63223503700000006, 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 761 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.369 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043607+0000", 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0299999999999998e-07 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043611+0000", 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3030000000000001e-06 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043621+0000", 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0023999999999999e-05 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043629+0000", 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8460000000000008e-06 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204276+0000", 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16064708899999999 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204439+0000", 2024-04-18T21:52:55.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016309500000000001 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.675841+0000", 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47140197700000003 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:770 3.13s1 3:cd8bc96a:::smithi09731216-48:head [read 0~1720320,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.055907+0000", 2024-04-18T21:52:55.373 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2257616090000001, 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41486172100000002, 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 770 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055907+0000", 2024-04-18T21:52:55.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055907+0000", 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055908+0000", 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2899999999999999e-06 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055918+0000", 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8970000000000003e-06 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055955+0000", 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7360000000000001e-05 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055971+0000", 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829e-05 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.061898+0000", 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059268000000000003 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.062275+0000", 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037725600000000002 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.470769+0000", 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40849328899999998 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.cs0 94/84 ECSubReadReply(tid=199, attrs_read=0))", 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.0774528449999998, 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46362988399999999, 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304179+0000", 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099963732 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304195+0000", 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5982999999999999e-05 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667760+0000", 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.36356425599999997 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667845+0000", 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5913000000000002e-05 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.cs0 94/84 ECSubRead(tid=200, to_read={3:3c73a4f8:::smithi09731216-45:head=0,356352,0}, subchunks={3:3c73a4f8:::smithi09731216-45:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.0769150139999999, 2024-04-18T21:52:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.463402286, 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304241+0000", 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099488015999999999 2024-04-18T21:52:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304269+0000", 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7447e-05 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667948+0000", 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.36367946200000001 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.668156+0000", 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020736100000000001 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.385 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:55.386 INFO:tasks.rados.rados.0.smithi097.stdout:325: finishing copy_from racing read to smithi09731216-5 2024-04-18T21:52:55.386 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is set 2024-04-18T21:52:55.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:52:55.387 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:55.383+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:55.487 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:55.482+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:55.512 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:55.512 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:55.512 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:55.512 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:55.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:52:55.583 INFO:tasks.rados.rados.0.smithi097.stdout:325: finishing copy_from to smithi09731216-5 2024-04-18T21:52:55.583 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 5 v 13 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-18T21:52:55.587 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:55.582+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:55.643 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:55.643 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:55.643 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:55.644 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:55.644 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:55.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:52:55.689 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:55.686+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:55.786 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:55.786 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:55.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:55.786 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:55.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.786 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:239 3.as0 3:5bb4af71:::smithi09731216-7:head [delete] snapc 4=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-18T21:52:55.786 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 61.769641092000001, 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40857630099999998, 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014034+0000", 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2500000000000001e-06 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014044+0000", 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1889999999999996e-06 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014067+0000", 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3251000000000002e-05 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114145+0000", 2024-04-18T21:52:55.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078193 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114306+0000", 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161044 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114320+0000", 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4044e-05 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317101+0000", 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20278075300000001 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317362+0000", 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026128799999999998 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.319467+0000", 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021048719999999998 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.326280+0000", 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006813375 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422400+0000", 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096119880000000005 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422608+0000", 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020816199999999999 2024-04-18T21:52:55.792 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:308 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~737280] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 57.693201016000003, 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69365002799999997, 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.793 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 308 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090475+0000", 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.976e-06 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091768+0000", 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00129317 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091803+0000", 2024-04-18T21:52:55.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4502999999999998e-05 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091826+0000", 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3235999999999999e-05 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270501+0000", 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.178674888 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270680+0000", 2024-04-18T21:52:55.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017917599999999999 2024-04-18T21:52:55.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.273556+0000", 2024-04-18T21:52:55.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028757610000000001 2024-04-18T21:52:55.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281580+0000", 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080240799999999994 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783886+0000", 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50230573000000001 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784122+0000", 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023650799999999999 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:309 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 57.691120023000003, 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69216249600000002, 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.799 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 309 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092555+0000", 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093122+0000", 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056744499999999997 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093146+0000", 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128999999999999e-05 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093165+0000", 2024-04-18T21:52:55.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8300000000000001e-05 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275352+0000", 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18218714899999999 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275632+0000", 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028059100000000001 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.277549+0000", 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001917087 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281584+0000", 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004034513 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784502+0000", 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50291762600000001 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784716+0000", 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021415599999999999 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:310 3.as0 3:5bb4af71:::smithi09731216-7:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 57.690478853000002, 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69187687499999995, 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.805 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 310 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093196+0000", 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093210+0000", 2024-04-18T21:52:55.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3517e-05 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093227+0000", 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7050000000000001e-05 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093243+0000", 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5965999999999999e-05 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279008+0000", 2024-04-18T21:52:55.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.185765398 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279287+0000", 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027941400000000002 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.280700+0000", 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412581 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482172+0000", 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20147153900000001 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784956+0000", 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30278430299999998 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785071+0000", 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115454 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.810 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:311 3.as0 3:5bb4af71:::smithi09731216-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 57.690403901000003, 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99553992999999996, 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 311 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.811 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093271+0000", 2024-04-18T21:52:55.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093278+0000", 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1459999999999999e-06 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093295+0000", 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7408999999999999e-05 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093328+0000", 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2276000000000001e-05 2024-04-18T21:52:55.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785901+0000", 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.692573628 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.786181+0000", 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027934999999999999 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.088809+0000", 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30262875700000003 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:429 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.337997465999997, 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51405278700000001, 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.815 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 429 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445677+0000", 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5260000000000001e-06 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445685+0000", 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445703+0000", 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8295999999999999e-05 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445718+0000", 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5383000000000002e-05 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545773+0000", 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100054752 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545785+0000", 2024-04-18T21:52:55.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1783000000000001e-05 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557166+0000", 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011381578 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557443+0000", 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027702800000000002 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.558675+0000", 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012320479999999999 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659884+0000", 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101208162 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959511+0000", 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29962765699999999 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959729+0000", 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021733700000000001 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:495 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~737280] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.276634033000001, 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.664354899, 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.821 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 495 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507043+0000", 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.596e-06 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508389+0000", 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013461460000000001 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508432+0000", 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3294999999999998e-05 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508456+0000", 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4134e-05 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581521+0000", 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.073064403 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581827+0000", 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030644999999999999 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.586475+0000", 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046472989999999997 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.699953+0000", 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.113478523 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171243+0000", 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47128973499999999 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171394+0000", 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000151318 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:496 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~573440] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.273446380000003, 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76183363900000001, 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 496 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510229+0000", 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.007e-06 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511015+0000", 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000785792 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511046+0000", 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0873000000000001e-05 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511061+0000", 2024-04-18T21:52:55.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5293000000000001e-05 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.691993+0000", 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.180931701 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.692191+0000", 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019870099999999999 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.694402+0000", 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022106690000000002 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.701975+0000", 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075735730000000001 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.271924+0000", 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56994823500000003 2024-04-18T21:52:55.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272061+0000", 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013679499999999999 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:497 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~196608] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.272397017999999, 2024-04-18T21:52:55.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76114050600000005, 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 497 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511278+0000", 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.387e-06 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511608+0000", 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033003500000000001 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511637+0000", 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9706e-05 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511661+0000", 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3439999999999999e-05 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696155+0000", 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18449448800000001 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696457+0000", 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000301496 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.698580+0000", 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021231879999999998 2024-04-18T21:52:55.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.702633+0000", 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040534079999999997 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272305+0000", 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56967170199999995 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272417+0000", 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111656 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:535 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~802816] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 30.385188305, 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60500271100000003, 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.838 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 535 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398487+0000", 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0650000000000001e-06 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399399+0000", 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091160899999999997 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399417+0000", 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7915999999999999e-05 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399426+0000", 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7070000000000004e-06 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.485946+0000", 2024-04-18T21:52:55.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086519677000000003 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.486140+0000", 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019389 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.488798+0000", 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002657707 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.600548+0000", 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11175003 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003305+0000", 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40275720799999998 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003488+0000", 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000182902 2024-04-18T21:52:55.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:536 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~638976] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 30.380210647999998, 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60040761799999998, 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.844 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 536 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:52:55.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.09e-07 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404080+0000", 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00061689199999999996 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404092+0000", 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2055000000000001e-05 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404101+0000", 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5650000000000002e-06 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491527+0000", 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087426312000000006 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491697+0000", 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017024799999999999 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.493862+0000", 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021641299999999998 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700761+0000", 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20689907199999999 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003718+0000", 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30295725099999998 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003870+0000", 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015228400000000001 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:537 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~131072] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 30.379240986999999, 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59990886700000001, 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.849 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 537 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:52:55.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404433+0000", 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.44e-07 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404610+0000", 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017705100000000001 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404620+0000", 2024-04-18T21:52:55.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8050000000000001e-06 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404625+0000", 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5029999999999997e-06 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495333+0000", 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.090708141000000006 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495532+0000", 2024-04-18T21:52:55.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001983 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.496836+0000", 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013041789999999999 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700772+0000", 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20393593900000001 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004176+0000", 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30340381399999999 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004341+0000", 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016549100000000001 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:538 3.10s0 3:0e754656:::smithi09731216-19:head [setxattr _header (57) in=7b] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:55.854 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 30.379038999999999, 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70001739100000004, 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 538 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404635+0000", 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6699999999999999e-07 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404637+0000", 2024-04-18T21:52:55.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.683e-06 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404642+0000", 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.967e-06 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404647+0000", 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.25e-06 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.497929+0000", 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.093281966999999993 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498095+0000", 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016633800000000001 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498905+0000", 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081039199999999995 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700775+0000", 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20186959700000001 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104337+0000", 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40356159400000002 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104652+0000", 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031513599999999999 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.859 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:539 3.10s0 3:0e754656:::smithi09731216-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 30.379020925999999, 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90384237700000003, 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 539 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404653+0000", 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0600000000000001e-07 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404655+0000", 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.95e-06 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404662+0000", 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9530000000000002e-06 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404666+0000", 2024-04-18T21:52:55.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4089999999999996e-06 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499383+0000", 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094717379000000004 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499540+0000", 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015672099999999999 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.105824+0000", 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60628387699999997 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.106088+0000", 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026422700000000002 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.308495+0000", 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20240645500000001 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:572 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~507904 in=507904b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.250451791, 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60481051399999997, 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 572 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.865 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533224+0000", 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.824e-06 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534169+0000", 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094509200000000002 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534205+0000", 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5482999999999999e-05 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534228+0000", 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2806000000000002e-05 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634530+0000", 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10030222799999999 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634798+0000", 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026831400000000002 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.638371+0000", 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035731249999999999 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744705+0000", 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106333831 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.137882+0000", 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39317701599999999 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.138032+0000", 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014979500000000001 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:573 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~606208 in=606208b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.249410482999998, 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70430669099999998, 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.870 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 573 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534265+0000", 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9800000000000001e-06 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535350+0000", 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010856869999999999 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535392+0000", 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1192999999999997e-05 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535427+0000", 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5559999999999998e-05 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642525+0000", 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107097461 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642617+0000", 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2181000000000004e-05 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.643719+0000", 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011017449999999999 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747184+0000", 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103465025 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238444+0000", 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49126059700000002 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238569+0000", 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012526200000000001 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:574 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~557056 in=557056b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:55.875 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.248215543000001, 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70342813999999998, 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 574 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:55.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535460+0000", 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.796e-06 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540737+0000", 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052778449999999998 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540774+0000", 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6372999999999999e-05 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540799+0000", 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5743999999999999e-05 2024-04-18T21:52:55.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744797+0000", 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20399719899999999 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744963+0000", 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016631799999999999 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.746246+0000", 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012830319999999999 2024-04-18T21:52:55.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747841+0000", 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001595352 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238775+0000", 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49093394899999998 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238886+0000", 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110532 2024-04-18T21:52:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:575 3.as0 3:561bcae7:::smithi09731216-2:head [setxattr _header (56) in=63b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.24283509, 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69838045999999998, 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:55.881 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 575 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540841+0000", 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4540000000000001e-06 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540850+0000", 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2329999999999995e-06 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540874+0000", 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3980000000000001e-05 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540893+0000", 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9173000000000001e-05 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748051+0000", 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20715831100000001 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748150+0000", 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9155999999999997e-05 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748574+0000", 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042389100000000002 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.753028+0000", 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044531889999999998 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.239064+0000", 2024-04-18T21:52:55.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.48603637100000002 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.239219+0000", 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154702 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 82/78 [PushReplyOp(3:0e754656:::smithi09731216-19:10)])", 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:52:55.886 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.488830821000001, 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81294819200000001, 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:52:55.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294850+0000", 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6299999999999998e-06 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294862+0000", 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.186e-05 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294892+0000", 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0103e-05 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294914+0000", 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1633000000000001e-05 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.107740+0000", 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81282626999999996 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.107791+0000", 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0695999999999997e-05 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 82/78 [PushReplyOp(3:0e754656:::smithi09731216-19:10)])", 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.487951677000002, 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91227883099999996, 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:55.890 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295723+0000", 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.703e-06 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295734+0000", 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.064e-05 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295762+0000", 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7606999999999998e-05 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:55.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295776+0000", 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.402e-05 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.207972+0000", 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91219607400000002 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.208000+0000", 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8787000000000001e-05 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.893 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:55.894 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:55.894 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:55.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:52:55.896 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:55.891+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:55.996 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:55.991+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:56.022 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:56.023 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:56.023 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:52:56.023 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:56.037 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:52:56.097 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:56.094+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:56.160 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:56.160 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:52:56.160 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:52:56.160 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:52:56.160 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:56.174 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:52:56.198 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:56.195+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 83.558205882999999, 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61088117099999995, 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.299 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 35 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735191+0000", 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3699999999999999e-07 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735266+0000", 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4856000000000003e-05 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735279+0000", 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.36e-05 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735288+0000", 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0860000000000008e-06 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242298+0000", 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50700964999999998 2024-04-18T21:52:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242568+0000", 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027023100000000001 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242974+0000", 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040533399999999998 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246576+0000", 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036026919999999998 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345792+0000", 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099215503999999996 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345953+0000", 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161189 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346071+0000", 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118192 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 83.558093389999996, 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61108955600000003, 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.305 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 36 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:56.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.23e-07 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735310+0000", 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7560000000000002e-06 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735323+0000", 2024-04-18T21:52:56.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735337+0000", 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4049999999999999e-05 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244243+0000", 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50890638099999996 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244422+0000", 2024-04-18T21:52:56.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000178495 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244638+0000", 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216562 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.247950+0000", 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033121750000000001 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346232+0000", 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098281564000000002 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346324+0000", 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2042999999999999e-05 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346392+0000", 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8052999999999994e-05 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.310 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 83.558044370999994, 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71224671699999997, 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 37 2024-04-18T21:52:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0999999999999998e-07 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735358+0000", 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2639999999999999e-06 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735367+0000", 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735375+0000", 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5360000000000004e-06 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245345+0000", 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50996983699999998 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245615+0000", 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027018099999999998 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246022+0000", 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000407394 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.248536+0000", 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025132330000000001 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447444+0000", 2024-04-18T21:52:56.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198908419 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447532+0000", 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8010000000000006e-05 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447598+0000", 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6383000000000005e-05 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 78.644643857999995, 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61732750999999997, 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 117 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648753+0000", 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:52:56.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648758+0000", 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648768+0000", 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.414e-06 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648777+0000", 2024-04-18T21:52:56.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.363e-06 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863479+0000", 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214701524 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863596+0000", 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117039 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863810+0000", 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021416800000000001 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.864846+0000", 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001036319 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265892+0000", 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40104641899999999 2024-04-18T21:52:56.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265995+0000", 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010209600000000001 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.266080+0000", 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5049000000000005e-05 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.322 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.987469597, 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60955713600000005, 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 161 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305927+0000", 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306056+0000", 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012895300000000001 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306068+0000", 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306077+0000", 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9430000000000006e-06 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412179+0000", 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106101709 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412267+0000", 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8251999999999995e-05 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412388+0000", 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121136 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420472+0000", 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080833510000000008 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915141+0000", 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49466944899999998 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915356+0000", 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215305 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915484+0000", 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012713500000000001 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.987305774000006, 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61031149600000001, 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.328 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 162 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306091+0000", 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306097+0000", 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6300000000000003e-06 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306106+0000", 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2199999999999998e-06 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306116+0000", 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8870000000000004e-06 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413869+0000", 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107753534 2024-04-18T21:52:56.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414135+0000", 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026615200000000002 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414518+0000", 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038217299999999997 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420492+0000", 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059740879999999998 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915940+0000", 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49544827600000002 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916143+0000", 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000202752 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916402+0000", 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025901100000000002 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.987269404000003, 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61064147300000005, 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.334 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 163 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.06e-07 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306136+0000", 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6999999999999997e-06 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306146+0000", 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8439999999999993e-06 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306154+0000", 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415593+0000", 2024-04-18T21:52:56.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109439392 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415833+0000", 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000240101 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416223+0000", 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000389804 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420508+0000", 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042851699999999996 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916537+0000", 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49602917499999999 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916669+0000", 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179899999999999 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916768+0000", 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8879e-05 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:56.340 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.987230870999994, 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61096294100000004, 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 164 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306166+0000", 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306275+0000", 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109749 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306287+0000", 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1182999999999999e-05 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306298+0000", 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1487e-05 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417044+0000", 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110745818 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417282+0000", 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000237712 2024-04-18T21:52:56.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417665+0000", 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000383194 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420523+0000", 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028578010000000001 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916880+0000", 2024-04-18T21:52:56.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49635731900000002 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917054+0000", 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017436200000000001 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917128+0000", 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3693000000000001e-05 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.346 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.987084875999997, 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70940268200000001, 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 165 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.347 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306312+0000", 2024-04-18T21:52:56.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306317+0000", 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0699999999999997e-06 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306327+0000", 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7030000000000001e-06 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306349+0000", 2024-04-18T21:52:56.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2160000000000001e-05 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419233+0000", 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112884159 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419499+0000", 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265865 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419851+0000", 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035253999999999998 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.421040+0000", 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011886380000000001 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015628+0000", 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59458825299999996 2024-04-18T21:52:56.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015677+0000", 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8470000000000002e-05 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015714+0000", 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7123999999999999e-05 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:438 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~622592] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.811321372000002, 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60190392599999998, 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 438 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.353 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482077+0000", 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3429999999999998e-06 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482901+0000", 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082364099999999996 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482933+0000", 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2128999999999997e-05 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482949+0000", 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5872999999999999e-05 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676460+0000", 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19351131799999999 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676647+0000", 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018723700000000001 2024-04-18T21:52:56.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.678595+0000", 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00194812 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.782086+0000", 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10349056299999999 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083745+0000", 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30165924199999999 2024-04-18T21:52:56.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083979+0000", 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023346 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:439 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~147456] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:56.358 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.809703886999998, 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70156516499999999, 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 439 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483694+0000", 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2129999999999998e-06 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483902+0000", 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020762600000000001 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483934+0000", 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2140000000000001e-05 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483950+0000", 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5792999999999999e-05 2024-04-18T21:52:56.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779688+0000", 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29573833799999999 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779887+0000", 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019900000000000001 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.781116+0000", 2024-04-18T21:52:56.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001229071 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.882567+0000", 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101451022 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185116+0000", 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.302548704 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185257+0000", 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014125800000000001 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:440 3.fs0 3:fcd1a062:::smithi09731216-16:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.809420011999997, 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.10743576, 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 440 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483977+0000", 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7000000000000003e-07 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483982+0000", 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4e-06 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483994+0000", 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.134e-05 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484003+0000", 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1169999999999993e-06 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186403+0000", 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70240069199999999 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186624+0000", 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000220523 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.187505+0000", 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088087199999999999 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188877+0000", 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001371723 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591138+0000", 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40226145699999999 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591412+0000", 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027366599999999999 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:441 3.fs0 3:fcd1a062:::smithi09731216-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:56.369 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.809380769000001, 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.413060682, 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 441 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484016+0000", 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8299999999999996e-07 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484019+0000", 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3069999999999998e-06 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484038+0000", 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978999999999999e-05 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484047+0000", 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5939999999999994e-06 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188125+0000", 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70407843599999997 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188355+0000", 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022941899999999999 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692687+0000", 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50433201000000005 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692919+0000", 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023258299999999999 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.897076+0000", 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20415657100000001 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.374 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:494 3.13s0 3:c8a99f7f:::smithi09731216-31:6 [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.797663802000002, 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59079312500000003, 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 494 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495734+0000", 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4929999999999999e-06 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495743+0000", 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5629999999999998e-06 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495768+0000", 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5147000000000002e-05 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495784+0000", 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6019e-05 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.596290+0000", 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100505504 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.596489+0000", 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019867499999999999 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886268+0000", 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.28977927399999998 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886434+0000", 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000166288 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.086525+0000", 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20009116199999999 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:563 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~491520] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:56.379 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.871998884, 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62112892600000003, 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 563 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421400+0000", 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8729999999999999e-06 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422217+0000", 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081729399999999998 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422245+0000", 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7497000000000001e-05 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422262+0000", 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7601999999999999e-05 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422362+0000", 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9270000000000003e-05 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422471+0000", 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109067 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.423731+0000", 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012607779999999999 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.425601+0000", 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018698549999999999 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042390+0000", 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61678826399999997 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042526+0000", 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013642600000000001 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.384 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:564 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~589824] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.861428871000001, 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61084178300000003, 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 564 2024-04-18T21:52:56.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431969+0000", 2024-04-18T21:52:56.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.64e-06 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432549+0000", 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00058043400000000001 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432569+0000", 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0203999999999998e-05 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432578+0000", 2024-04-18T21:52:56.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4619999999999996e-06 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432649+0000", 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0669999999999999e-05 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432762+0000", 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113474 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.433682+0000", 2024-04-18T21:52:56.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092051399999999999 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.434979+0000", 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012963530000000001 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042699+0000", 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60771967500000001 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042809+0000", 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110357 2024-04-18T21:52:56.389 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:565 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~557056] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.851337949000001, 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60103031100000004, 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.390 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 565 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:52:56.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442060+0000", 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.843e-06 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442728+0000", 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00066799899999999996 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442745+0000", 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6932999999999999e-05 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442753+0000", 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8029999999999997e-06 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.537940+0000", 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.095187356000000001 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.538060+0000", 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119986 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.539336+0000", 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012755990000000001 2024-04-18T21:52:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546080+0000", 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067440950000000003 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042980+0000", 2024-04-18T21:52:56.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49690054099999997 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.043088+0000", 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108156 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:566 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~16384] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.850628665999999, 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60493431399999997, 2024-04-18T21:52:56.396 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 566 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442768+0000", 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5899999999999995e-07 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442790+0000", 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1687000000000001e-05 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442802+0000", 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2564e-05 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442808+0000", 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2830000000000001e-06 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540321+0000", 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097512961999999995 2024-04-18T21:52:56.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540613+0000", 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029227700000000002 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.542271+0000", 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016580010000000001 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546084+0000", 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038128620000000002 2024-04-18T21:52:56.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.047609+0000", 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50152518300000004 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.047702+0000", 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2736000000000001e-05 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:567 3.13s0 3:cd8bc96a:::smithi09731216-48:head [setxattr _header (57) in=7b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.850579201999999, 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70082249600000002, 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 567 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9399999999999995e-07 2024-04-18T21:52:56.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442820+0000", 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4229999999999999e-06 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442826+0000", 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0440000000000001e-06 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442830+0000", 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0640000000000004e-06 2024-04-18T21:52:56.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.543400+0000", 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100569913 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.543930+0000", 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052991099999999999 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.545388+0000", 2024-04-18T21:52:56.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014587059999999999 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.646614+0000", 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101225551 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.143425+0000", 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.496811477 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.143639+0000", 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021391300000000001 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:568 3.13s0 3:cd8bc96a:::smithi09731216-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.850560956999999, 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90286167399999995, 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:52:56.407 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 568 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:52:56.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999998e-07 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442837+0000", 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8020000000000001e-06 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:52:56.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442844+0000", 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5869999999999998e-06 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442848+0000", 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9929999999999997e-06 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.545932+0000", 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103084052 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546068+0000", 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013653800000000001 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.144727+0000", 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59865855499999998 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.144992+0000", 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026491399999999999 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:52:56.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.345697+0000", 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200704883 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:52:56.412 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:52:56.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:52:56.414 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:52:56.410+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:52:56.515 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:56.510+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 100/98 ECSubWriteReply(tid=16, last_complete=99'12, committed=1, applied=1))", 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:56.409747+0000", 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.13699255599999999, 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.137029279, 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.409747+0000", 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.409747+0000", 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.409748+0000", 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3060000000000001e-06 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.409757+0000", 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9959999999999999e-06 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.409784+0000", 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6811000000000001e-05 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.409801+0000", 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6912e-05 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 100/98 ECSubWriteReply(tid=16, last_complete=99'12, committed=1, applied=1))", 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:56.505734+0000", 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.041005554, 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041124255999999998, 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.505734+0000", 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.505734+0000", 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.505735+0000", 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.813e-06 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.505747+0000", 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1086e-05 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.505784+0000", 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7246999999999998e-05 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.505803+0000", 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9293999999999999e-05 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.17s0 100/98 ECSubWriteReply(tid=16, last_complete=99'12, committed=1, applied=1))", 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:56.509488+0000", 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.037251332999999998, 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037442507999999999, 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.509488+0000", 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.509488+0000", 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.509489+0000", 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5060000000000001e-06 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.509499+0000", 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0929999999999992e-06 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.509526+0000", 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7220999999999999e-05 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.509545+0000", 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8865e-05 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-04-18T21:52:56.555 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:56.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:52:56.616 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:56.610+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:56.680 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:56.680 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:56.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:52:56.716 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:56.710+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:56.817 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:56.815+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 30/26 ECSubWrite(tid=8, reqid=client.4509.0:95, at_version=30'6, trim_to=0'0, roll_forward_to=29'4))", 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 70.252360187999997, 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201960743, 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.821 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566448+0000", 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566449+0000", 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.77e-06 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566495+0000", 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5488999999999998e-05 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.566536+0000", 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0756000000000001e-05 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.666638+0000", 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102466 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766859+0000", 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10022078500000001 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.766875+0000", 2024-04-18T21:52:56.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5806000000000001e-05 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768312+0000", 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014373389999999999 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.768408+0000", 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6331999999999999e-05 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:118 3.17s0 3:eda737c1:::smithi09731216-40:head [append 0~671744,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 69.214116958999995, 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1801127490000001, 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 118 2024-04-18T21:52:56.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604691+0000", 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.604693+0000", 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2970000000000002e-06 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.605995+0000", 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001302276 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606021+0000", 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5320000000000002e-05 2024-04-18T21:52:56.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606036+0000", 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5133000000000001e-05 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606232+0000", 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000196672 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606237+0000", 2024-04-18T21:52:56.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7199999999999997e-06 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484281+0000", 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.87804420999999999 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.484592+0000", 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031024299999999998 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.487837+0000", 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003245571 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490891+0000", 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030539719999999998 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:56.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784613+0000", 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.293722237 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.784804+0000", 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019009800000000001 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:119 3.17s0 3:eda737c1:::smithi09731216-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:56.832 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 69.212735109999997, 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.179191986, 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 119 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606073+0000", 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606074+0000", 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.257e-06 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606080+0000", 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0100000000000001e-06 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606095+0000", 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5413e-05 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606104+0000", 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0899999999999994e-06 2024-04-18T21:52:56.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606286+0000", 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000181552 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606287+0000", 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.026e-06 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493720+0000", 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.88743291700000004 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.493872+0000", 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015190200000000001 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495103+0000", 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00123149 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.496855+0000", 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017518189999999999 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785096+0000", 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.28824129900000001 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785265+0000", 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016821100000000001 2024-04-18T21:52:56.838 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:120 3.17s0 3:eda737c1:::smithi09731216-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 69.212665979999997, 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4107226310000001, 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:56.839 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 120 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606142+0000", 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606143+0000", 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.16e-07 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606148+0000", 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4569999999999996e-06 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606163+0000", 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503e-05 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.606175+0000", 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2036e-05 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706293+0000", 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100117875 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.706297+0000", 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.126e-06 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495621+0000", 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78932349599999996 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.495757+0000", 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013642600000000001 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.785995+0000", 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29023745499999998 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:56.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.786127+0000", 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013212200000000001 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:49.016864+0000", 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.23073769199999999 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.17s0 32/31 ECSubRead(tid=3, to_read={3:eda737c1:::smithi09731216-40:head=0,4096,0}, subchunks={3:eda737c1:::smithi09731216-40:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:56.844 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 68.031624979, 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.210779366, 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787183+0000", 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.787250+0000", 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7152999999999999e-05 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.887353+0000", 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100102647 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997704+0000", 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110351217 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.997962+0000", 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000258349 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.0s5 41/31 ECSubWrite(tid=28, reqid=client.4509.0:304, at_version=41'12, trim_to=0'0, roll_forward_to=32'8))", 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.649440411999997, 2024-04-18T21:52:56.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201742379, 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169367+0000", 2024-04-18T21:52:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169369+0000", 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6339999999999999e-06 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169413+0000", 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3860999999999999e-05 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.169487+0000", 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4578000000000003e-05 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.269583+0000", 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100095613 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369850+0000", 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10026700299999999 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.369869+0000", 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9114e-05 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371022+0000", 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001152943 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.371110+0000", 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7633000000000006e-05 2024-04-18T21:52:56.852 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.bs6 41/26 ECSubWrite(tid=56, reqid=osd.10.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'25))", 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 58.033364388000003, 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201706881, 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785443+0000", 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785445+0000", 2024-04-18T21:52:56.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7999999999999999e-06 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785499+0000", 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3730999999999998e-05 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785543+0000", 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3961000000000001e-05 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.785568+0000", 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.561e-05 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.985970+0000", 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200401511 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.986000+0000", 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0224000000000001e-05 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987085+0000", 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001085145 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.987150+0000", 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4899000000000003e-05 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.2s5 58/31 ECSubWrite(tid=87, reqid=client.4509.0:422, at_version=58'6, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 42.160621847000002, 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201964437, 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658186+0000", 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658187+0000", 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5069999999999999e-06 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658288+0000", 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010070500000000001 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658315+0000", 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7348e-05 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658331+0000", 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5559000000000001e-05 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.758532+0000", 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100200734 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.758549+0000", 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6823000000000001e-05 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860062+0000", 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101513567 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860150+0000", 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8194000000000003e-05 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.2s5 58/31 ECSubWrite(tid=88, reqid=client.4509.0:423, at_version=58'7, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 42.160437588999997, 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20184664099999999, 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658370+0000", 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658371+0000", 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5400000000000001e-07 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658390+0000", 2024-04-18T21:52:56.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8907e-05 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658404+0000", 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3974999999999999e-05 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.658414+0000", 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0156000000000001e-05 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.759248+0000", 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10083389299999999 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.759263+0000", 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5101e-05 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860159+0000", 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10089624599999999 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.860217+0000", 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7809000000000002e-05 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.fs7 62/26 ECSubWrite(tid=203, reqid=osd.7.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=62'26))", 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 38.242601129999997, 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20165132299999999, 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.867 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576207+0000", 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576209+0000", 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9300000000000002e-06 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576234+0000", 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5287999999999999e-05 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.576260+0000", 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5808000000000001e-05 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.676352+0000", 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10009247 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.776560+0000", 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100207532 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.776579+0000", 2024-04-18T21:52:56.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8936e-05 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.777791+0000", 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00121206 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:18.777858+0000", 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7298999999999994e-05 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 64/26 ECSubWrite(tid=97, reqid=osd.5.0:97, at_version=64'55, trim_to=0'0, roll_forward_to=62'51))", 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 36.220623273000001, 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.202670817, 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598184+0000", 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598185+0000", 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598203+0000", 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8168e-05 2024-04-18T21:52:56.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.598214+0000", 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0509999999999999e-05 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.698303+0000", 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100089224 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.798517+0000", 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100213922 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.798550+0000", 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3120999999999997e-05 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.800786+0000", 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002236334 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.800855+0000", 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8966000000000003e-05 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/54 ECSubWrite(tid=8, reqid=osd.9.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=64'13))", 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 35.935795867000003, 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30101452299999998, 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.876 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883012+0000", 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883014+0000", 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3800000000000001e-06 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883043+0000", 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8924999999999998e-05 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883085+0000", 2024-04-18T21:52:56.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1674000000000001e-05 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.883109+0000", 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4345000000000002e-05 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.083392+0000", 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200282457 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.083411+0000", 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9156000000000001e-05 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.183969+0000", 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100558567 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.184026+0000", 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7018999999999997e-05 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.880 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.cs1 64/26 ECSubWrite(tid=99, reqid=osd.5.0:97, at_version=0'0, trim_to=0'0, roll_forward_to=64'55))", 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 35.913050331000001, 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201502976, 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.881 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905757+0000", 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905758+0000", 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.91e-07 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905764+0000", 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1399999999999997e-06 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905769+0000", 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5029999999999997e-06 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:20.905776+0000", 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6989999999999997e-06 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.106403+0000", 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20062649299999999 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.106420+0000", 2024-04-18T21:52:56.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7187999999999999e-05 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.107159+0000", 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00073935799999999996 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:21.107260+0000", 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101304 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.885 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es4 68/26 ECSubWrite(tid=58, reqid=osd.2.0:58, at_version=68'44, trim_to=0'0, roll_forward_to=66'40))", 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.436732978000002, 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.205606013, 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:56.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382075+0000", 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382077+0000", 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.863e-06 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382131+0000", 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.465e-05 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.382164+0000", 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3081999999999999e-05 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.482269+0000", 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10010452 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.582575+0000", 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100306485 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.582604+0000", 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8462e-05 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.587585+0000", 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049807999999999996 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.587681+0000", 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6150999999999995e-05 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s3 68/58 ECSubWrite(tid=215, reqid=osd.7.0:214, at_version=0'0, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 31.898538108, 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20176354299999999, 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.890 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920270+0000", 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920271+0000", 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8530000000000001e-06 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920293+0000", 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1926e-05 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:24.920324+0000", 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1047999999999998e-05 2024-04-18T21:52:56.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.020468+0000", 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100143498 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.120740+0000", 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100271761 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.120767+0000", 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7268999999999999e-05 2024-04-18T21:52:56.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.121955+0000", 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011884599999999999 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:25.122033+0000", 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7727999999999998e-05 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.es4 73/72 ECSubWrite(tid=63, reqid=client.4509.0:562, at_version=73'48, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.022380+0000", 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.796428098, 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20326635300000001, 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022380+0000", 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022380+0000", 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022381+0000", 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022429+0000", 2024-04-18T21:52:56.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8031000000000001e-05 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022462+0000", 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2555000000000003e-05 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.022479+0000", 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6806000000000001e-05 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.222841+0000", 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200362389 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.222877+0000", 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6405000000000001e-05 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.225557+0000", 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026795249999999999 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.225646+0000", 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8988999999999998e-05 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s3 74/58 ECSubWrite(tid=222, reqid=client.4509.0:566, at_version=74'62, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.544653+0000", 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.274154943999999, 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.203329066, 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.899 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544653+0000", 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544653+0000", 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544654+0000", 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2299999999999999e-07 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544720+0000", 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6229999999999994e-05 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.544748+0000", 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7702e-05 2024-04-18T21:52:56.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.644842+0000", 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100094432 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.745084+0000", 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100242131 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.745111+0000", 2024-04-18T21:52:56.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6418999999999999e-05 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.747885+0000", 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027739969999999998 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.747982+0000", 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7231999999999993e-05 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.903 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "pg_update_log_missing(3.13s3 epoch 79/78 rep_tid 234 entries 79'66 (0'0) error 3:c8da3643:::smithi09731216-25:head by client.4509.0:610 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:35.805921+0000", 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 21.012887057, 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20149904199999999, 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.805921+0000", 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.805921+0000", 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.805922+0000", 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3829999999999999e-06 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.805956+0000", 2024-04-18T21:52:56.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3958999999999998e-05 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.806003+0000", 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6752000000000002e-05 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:35.906116+0000", 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10011363400000001 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:36.006389+0000", 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10027270100000001 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:36.007420+0000", 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010306130000000001 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 95/90 ECSubWrite(tid=27, reqid=client.4509.0:764, at_version=95'24, trim_to=0'0, roll_forward_to=94'23))", 2024-04-18T21:52:56.907 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.718860+0000", 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.0999481930000004, 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20340918299999999, 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.718860+0000", 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.718860+0000", 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.718860+0000", 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2999999999999999e-07 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.718983+0000", 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012280099999999999 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.718998+0000", 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.452e-05 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.819106+0000", 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10010880799999999 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.819339+0000", 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000232067 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.819371+0000", 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1955000000000002e-05 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.922214+0000", 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10284386399999999 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.922269+0000", 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4338000000000001e-05 2024-04-18T21:52:56.911 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s2 99/97 ECSubWrite(tid=39, reqid=osd.9.0:39, at_version=99'28, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:55.707823+0000", 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.1109849300000001, 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30199320499999999, 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.707823+0000", 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.707823+0000", 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.707825+0000", 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-06 2024-04-18T21:52:56.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.707879+0000", 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4752e-05 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.707912+0000", 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2647000000000001e-05 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:56.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.808008+0000", 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100096166 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.008319+0000", 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20031104 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.008337+0000", 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7591000000000001e-05 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.009731+0000", 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001394574 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.009816+0000", 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4544999999999998e-05 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:56.916 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:56.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:52:56.918 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:56.914+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:56.994 INFO:tasks.rados.rados.0.smithi097.stdout:316: done (2 left) 2024-04-18T21:52:56.994 INFO:tasks.rados.rados.0.smithi097.stdout:325: done (1 left) 2024-04-18T21:52:56.994 INFO:tasks.rados.rados.0.smithi097.stdout:327: done (0 left) 2024-04-18T21:52:56.995 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 38 current snap is 24 2024-04-18T21:52:56.995 INFO:tasks.rados.rados.0.smithi097.stdout:328: seq_num 130 ranges {2752512=638976,3391488=671744,4063232=802816,4866048=81920} 2024-04-18T21:52:57.019 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:57.015+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:57.048 INFO:tasks.rados.rados.0.smithi097.stdout:328: writing smithi09731216-38 from 2752512 to 3391488 tid 1 2024-04-18T21:52:57.056 INFO:tasks.rados.rados.0.smithi097.stdout:328: writing smithi09731216-38 from 3391488 to 4063232 tid 2 2024-04-18T21:52:57.065 INFO:tasks.rados.rados.0.smithi097.stdout:328: writing smithi09731216-38 from 4063232 to 4866048 tid 3 2024-04-18T21:52:57.066 INFO:tasks.rados.rados.0.smithi097.stdout:328: writing smithi09731216-38 from 4866048 to 4947968 tid 4 2024-04-18T21:52:57.066 INFO:tasks.rados.rados.0.smithi097.stdout:329: read oid 5 snap -1 2024-04-18T21:52:57.067 INFO:tasks.rados.rados.0.smithi097.stdout:329: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-18T21:52:57.067 INFO:tasks.rados.rados.0.smithi097.stdout:330: rollback oid 41 current snap is 24 2024-04-18T21:52:57.067 INFO:tasks.rados.rados.0.smithi097.stdout:330: done (2 left) 2024-04-18T21:52:57.067 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 46 current snap is 24 2024-04-18T21:52:57.067 INFO:tasks.rados.rados.0.smithi097.stdout:331: seq_num 131 ranges {770048=655360,1425408=425984,1851392=638976,2490368=491520} 2024-04-18T21:52:57.078 INFO:tasks.rados.rados.0.smithi097.stdout:331: writing smithi09731216-46 from 770048 to 1425408 tid 1 2024-04-18T21:52:57.081 INFO:tasks.rados.rados.0.smithi097.stdout:331: writing smithi09731216-46 from 1425408 to 1851392 tid 2 2024-04-18T21:52:57.087 INFO:tasks.rados.rados.0.smithi097.stdout:331: writing smithi09731216-46 from 1851392 to 2490368 tid 3 2024-04-18T21:52:57.090 INFO:tasks.rados.rados.0.smithi097.stdout:331: writing smithi09731216-46 from 2490368 to 2981888 tid 4 2024-04-18T21:52:57.090 INFO:tasks.rados.rados.0.smithi097.stdout:332: setattr oid 40 current snap is 24 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpRead(3.9s5 100/90 ECSubRead(tid=180, to_read={3:9941efaf:::smithi09731216-5:head=0,110592,0}, subchunks={3:9941efaf:::smithi09731216-5:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:57.070520+0000", 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.023235542000000001, 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.023282002, 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:57.093 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.070520+0000", 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.070520+0000", 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.070521+0000", 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3659999999999999e-06 2024-04-18T21:52:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.070538+0000", 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6796e-05 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.070569+0000", 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0987e-05 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:57.096 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 1 2024-04-18T21:52:57.096 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:57.096 INFO:tasks.rados.rados.0.smithi097.stdout:333: delete oid 33 current snap is 24 2024-04-18T21:52:57.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:52:57.120 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:57.114+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:57.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:52:57.220 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:57.215+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:57.280 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:57.280 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:57.280 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:57.280 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:57.281 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:57.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:333: done (4 left) 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:334: read oid 17 snap -1 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:334: expect (ObjNum 113 snap 23 seq_num 113) 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:335: read oid 1 snap -1 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:335: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:336: copy_from oid 16 from oid 50 current snap is 24 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:337: rmattr oid 8 current snap is 24 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:337: done (7 left) 2024-04-18T21:52:57.300 INFO:tasks.rados.rados.0.smithi097.stdout:338: delete oid 41 current snap is 24 2024-04-18T21:52:57.321 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:57.318+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:57.359 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:52:57.359 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:52:57.368 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 40 v 13 (ObjNum 65 snap 0 seq_num 1485210488) dirty exists 2024-04-18T21:52:57.391 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:52:57.391 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:52:57.391 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-18T21:52:57.391 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-18T21:52:57.391 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-18T21:52:57.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-04-18T21:52:57.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:52:57.422 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:52:57.422 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:52:57.423 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:52:57.418+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:52:57.439 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:52:57.439 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:52:57.466 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:57.466 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:57.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 73/58 ECSubWrite(tid=218, reqid=client.4509.0:563, at_version=73'59, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.039122605999999, 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30546464400000001, 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424972+0000", 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.424974+0000", 2024-04-18T21:52:57.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9e-06 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425568+0000", 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00059416700000000005 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425597+0000", 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9305000000000001e-05 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.425613+0000", 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5449000000000001e-05 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.625991+0000", 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.200377891 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.626011+0000", 2024-04-18T21:52:57.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0956000000000001e-05 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730378+0000", 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104366235 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730436+0000", 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8740999999999999e-05 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.13s7 73/58 ECSubWrite(tid=219, reqid=client.4509.0:564, at_version=73'60, trim_to=0'0, roll_forward_to=68'57))", 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 27.028783481000001, 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29521209100000001, 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435311+0000", 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435313+0000", 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.807e-06 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435737+0000", 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042456200000000002 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.435778+0000", 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0555e-05 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.536874+0000", 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101095886 2024-04-18T21:52:57.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.727552+0000", 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19067846699999999 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.727571+0000", 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8941999999999999e-05 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730493+0000", 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029220800000000001 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:30.730523+0000", 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9791999999999999e-05 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:731 3.12s0 3:4c1ede14:::smithi09731216-49:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.487911+0000", 2024-04-18T21:52:57.476 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.9761837959999999, 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.316265252, 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 731 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487911+0000", 2024-04-18T21:52:57.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487911+0000", 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487913+0000", 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2919999999999998e-06 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487921+0000", 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4749999999999993e-06 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487946+0000", 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4958999999999999e-05 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487958+0000", 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1948e-05 2024-04-18T21:52:57.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488076+0000", 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117897 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488092+0000", 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5722e-05 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.498251+0000", 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010159559 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.498408+0000", 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015634399999999999 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.498960+0000", 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00055209400000000004 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.503027+0000", 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040673569999999997 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.803500+0000", 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30047284800000001 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.804176+0000", 2024-04-18T21:52:57.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067575700000000003 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:732 3.12s0 3:4c1ede14:::smithi09731216-49:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.9761059589999999, 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.417434898, 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 732 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487988+0000", 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487989+0000", 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8299999999999997e-07 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.487992+0000", 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.253e-06 2024-04-18T21:52:57.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488000+0000", 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7090000000000002e-06 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488006+0000", 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3550000000000001e-06 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488154+0000", 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014786700000000001 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.488167+0000", 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2836e-05 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.499213+0000", 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011045981999999999 2024-04-18T21:52:57.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.499412+0000", 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000198519 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.500214+0000", 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00080208099999999995 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.503072+0000", 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00285819 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.904913+0000", 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40184145999999998 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:50.905423+0000", 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00051006300000000005 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:762 3.12s0 3:4c1ede14:::smithi09731216-49:head [append 0~491520] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.047027+0000", 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.4170670479999998, 2024-04-18T21:52:57.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.56884067100000002, 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 762 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.047027+0000", 2024-04-18T21:52:57.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.047027+0000", 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.047030+0000", 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6800000000000002e-06 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.047822+0000", 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00079196699999999997 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.047855+0000", 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3385e-05 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.047873+0000", 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7382999999999999e-05 2024-04-18T21:52:57.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.204670+0000", 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15679752199999999 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.204949+0000", 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027850999999999999 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.206605+0000", 2024-04-18T21:52:57.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016563820000000001 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.413393+0000", 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20678732899999999 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.615258+0000", 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.201865985 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.615868+0000", 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00060952800000000002 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:763 3.12s0 3:4c1ede14:::smithi09731216-49:head [append 0~802816] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.051684+0000", 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.4124103479999999, 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.56490670300000001, 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 763 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.051684+0000", 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.051684+0000", 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.051687+0000", 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1599999999999998e-06 2024-04-18T21:52:57.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.052782+0000", 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010950420000000001 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.052820+0000", 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8099000000000002e-05 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.052845+0000", 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4652999999999999e-05 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.207446+0000", 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15460104199999999 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.208041+0000", 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00059532900000000002 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.211607+0000", 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035661159999999998 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.413531+0000", 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20192389999999999 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616276+0000", 2024-04-18T21:52:57.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20274434799999999 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616591+0000", 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031501400000000002 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:764 3.12s0 3:4c1ede14:::smithi09731216-49:head [append 0~720896] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.056309+0000", 2024-04-18T21:52:57.502 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.4077858279999997, 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1730276589999999, 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 764 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.056309+0000", 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.056309+0000", 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.056311+0000", 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5560000000000002e-06 2024-04-18T21:52:57.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057742+0000", 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001431208 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057776+0000", 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3198999999999998e-05 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057792+0000", 2024-04-18T21:52:57.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6234000000000001e-05 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.617152+0000", 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.55936042600000002 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.617331+0000", 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000179054 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.618662+0000", 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013309400000000001 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.622397+0000", 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003734387 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.228599+0000", 2024-04-18T21:52:57.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.60620201299999998 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.229336+0000", 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00073764200000000005 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:765 3.12s0 3:4c1ede14:::smithi09731216-49:head [append 0~32768] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.057820+0000", 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.4062745290000001, 2024-04-18T21:52:57.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1735478660000001, 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 765 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057820+0000", 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057820+0000", 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057821+0000", 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3230000000000001e-06 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057860+0000", 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8872999999999999e-05 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057874+0000", 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4432e-05 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057888+0000", 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3703000000000001e-05 2024-04-18T21:52:57.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.619277+0000", 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.56138915599999994 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.619490+0000", 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021254499999999999 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.620454+0000", 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00096385500000000001 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.622398+0000", 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001944482 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.230644+0000", 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.60824565399999997 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.231368+0000", 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00072384299999999995 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:766 3.12s0 3:4c1ede14:::smithi09731216-49:head [setxattr _header (57) in=7b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.057905+0000", 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.406189361, 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.174494216, 2024-04-18T21:52:57.514 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 766 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057905+0000", 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057905+0000", 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057906+0000", 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3600000000000002e-07 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057912+0000", 2024-04-18T21:52:57.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8830000000000003e-06 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057923+0000", 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1303e-05 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.057953+0000", 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0275999999999999e-05 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.620967+0000", 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.56301348100000004 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.621165+0000", 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019839 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.621957+0000", 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00079204399999999999 2024-04-18T21:52:57.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.622511+0000", 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00055371200000000004 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.231651+0000", 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.60913959500000003 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.232399+0000", 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00074859600000000003 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=23, last_complete=94'21, committed=1, applied=1))", 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.105421+0000", 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.3586736540000004, 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20771136900000001, 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.520 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.105421+0000", 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.105421+0000", 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.105422+0000", 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.187e-06 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.105429+0000", 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.759e-06 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.105446+0000", 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6959000000000001e-05 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.105456+0000", 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.007e-05 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.313034+0000", 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.207578031 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.313132+0000", 2024-04-18T21:52:57.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8363000000000007e-05 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=25, last_complete=94'23, committed=1, applied=1))", 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.222068+0000", 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.242026675, 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29455472700000002, 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.524 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.222068+0000", 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.222068+0000", 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.222068+0000", 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4399999999999999e-07 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.222073+0000", 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6070000000000001e-06 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.222088+0000", 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4885000000000001e-05 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.222101+0000", 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2841e-05 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.516581+0000", 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.294480413 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.516622+0000", 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1236999999999998e-05 2024-04-18T21:52:57.527 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=24, last_complete=94'22, committed=1, applied=1))", 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.264362+0000", 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.1997326939999997, 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.24988175600000001, 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264362+0000", 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264362+0000", 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264364+0000", 2024-04-18T21:52:57.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1940000000000001e-06 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264375+0000", 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1175999999999999e-05 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264408+0000", 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2889000000000003e-05 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264428+0000", 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0305999999999998e-05 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.514186+0000", 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.24975777599999999 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.514243+0000", 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7414999999999998e-05 2024-04-18T21:52:57.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=25, last_complete=94'23, committed=1, applied=1))", 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.264771+0000", 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.1993238059999998, 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.35122914999999999, 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264771+0000", 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264771+0000", 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264772+0000", 2024-04-18T21:52:57.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.127e-06 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264780+0000", 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9370000000000004e-06 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264805+0000", 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5792e-05 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.264821+0000", 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5543e-05 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.615961+0000", 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.35113980500000003 2024-04-18T21:52:57.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616000+0000", 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8946000000000003e-05 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=25, last_complete=94'23, committed=1, applied=1))", 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.275985+0000", 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.1881095259999999, 2024-04-18T21:52:57.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.24055966300000001, 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.275985+0000", 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.275985+0000", 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.275986+0000", 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0699999999999996e-07 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.275990+0000", 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7360000000000001e-06 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.276006+0000", 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5950000000000001e-05 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.276016+0000", 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2359999999999993e-06 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.516506+0000", 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.24048991 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.516545+0000", 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9023999999999998e-05 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=24, last_complete=94'22, committed=1, applied=1))", 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.308382+0000", 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.1557124999999999, 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20792762100000001, 2024-04-18T21:52:57.540 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.308382+0000", 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.308382+0000", 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.308384+0000", 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1299999999999999e-06 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.308394+0000", 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8970000000000003e-06 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.308421+0000", 2024-04-18T21:52:57.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7402e-05 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408538+0000", 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10011666600000001 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.516269+0000", 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.107730624 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.516310+0000", 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0902000000000002e-05 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.12s0 94/90 ECSubWriteReply(tid=25, last_complete=94'23, committed=1, applied=1))", 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.408579+0000", 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.0555154770000001, 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20750092000000001, 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.544 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408579+0000", 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408579+0000", 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408580+0000", 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4499999999999995e-07 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408587+0000", 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.452e-06 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408602+0000", 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4232e-05 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408609+0000", 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6469999999999998e-06 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616045+0000", 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20743553000000001 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616080+0000", 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5114000000000003e-05 2024-04-18T21:52:57.547 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.3s1 94/93 ECSubWrite(tid=312, reqid=client.4509.0:729, at_version=94'155, trim_to=0'0, roll_forward_to=0'0))", 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.408814+0000", 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.0552803190000004, 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.25769818500000002, 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408814+0000", 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408814+0000", 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408814+0000", 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3000000000000002e-07 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408847+0000", 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2948999999999999e-05 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.408862+0000", 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4803e-05 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.508955+0000", 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10009334 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.564725+0000", 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.055769785000000002 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.564751+0000", 2024-04-18T21:52:57.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5675999999999999e-05 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.666394+0000", 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101643522 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.666512+0000", 2024-04-18T21:52:57.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011778 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.3s1 99/93 ECSubWrite(tid=350, reqid=osd.4.0:349, at_version=0'0, trim_to=0'0, roll_forward_to=98'187))", 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:55.809997+0000", 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.654097377, 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30319997700000001, 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.553 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.809997+0000", 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.809997+0000", 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.809998+0000", 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.23e-07 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.810015+0000", 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7351999999999999e-05 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.810031+0000", 2024-04-18T21:52:57.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5968e-05 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.810044+0000", 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2952999999999999e-05 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.910210+0000", 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10016628499999999 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.910223+0000", 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2143e-05 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.113110+0000", 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20288714399999999 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.113197+0000", 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7309e-05 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.557 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.3s1 99/93 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=99'189, trim_to=0'0, roll_forward_to=98'187))", 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:55.810195+0000", 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.6538992430000001, 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30307423900000002, 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.558 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.810195+0000", 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.810195+0000", 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.810196+0000", 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3000000000000001e-07 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.810230+0000", 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3946999999999999e-05 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.810245+0000", 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5322e-05 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:55.910364+0000", 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10011956399999999 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.011453+0000", 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10108848600000001 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.011476+0000", 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2708e-05 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.113212+0000", 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10173668700000001 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.113269+0000", 2024-04-18T21:52:57.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7095000000000001e-05 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.3s1 99/93 ECSubWrite(tid=353, reqid=osd.4.0:352, at_version=0'0, trim_to=0'0, roll_forward_to=99'191))", 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:56.195348+0000", 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.268746749, 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.20580992000000001, 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.563 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.195348+0000", 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.195348+0000", 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.195349+0000", 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.42e-06 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.195372+0000", 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2745e-05 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.195399+0000", 2024-04-18T21:52:57.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7203999999999999e-05 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.195423+0000", 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4043999999999999e-05 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.295635+0000", 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10021205599999999 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.295651+0000", 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5764999999999999e-05 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.401020+0000", 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105369449 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:56.401158+0000", 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013723699999999999 2024-04-18T21:52:57.567 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.568 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.568 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.568 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.568 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.568 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:57.569 INFO:tasks.rados.rados.0.smithi097.stdout:328: finishing write tid 1 to smithi09731216-38 2024-04-18T21:52:57.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:52:57.571 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:52:57.571 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-18T21:52:57.571 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:52:57.571 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:57.566+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:57.619 INFO:tasks.rados.rados.0.smithi097.stdout:328: finishing write tid 2 to smithi09731216-38 2024-04-18T21:52:57.633 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:52:57.634 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:52:57.670 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:57.666+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:57.701 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:57.701 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:57.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.701 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.8s1 100/86 ECSubWrite(tid=186, reqid=client.4509.0:829, at_version=100'43, trim_to=0'0, roll_forward_to=98'38))", 2024-04-18T21:52:57.701 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:57.610730+0000", 2024-04-18T21:52:57.701 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.091128331000000007, 2024-04-18T21:52:57.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.091168094000000005, 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "no flag points reached", 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.610730+0000", 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.610730+0000", 2024-04-18T21:52:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.610731+0000", 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1299999999999998e-07 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.610756+0000", 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5282e-05 2024-04-18T21:52:57.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.610772+0000", 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5566999999999999e-05 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.5s2 101/98 ECSubWrite(tid=228, reqid=osd.5.0:227, at_version=0'0, trim_to=0'0, roll_forward_to=100'42))", 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:57.612043+0000", 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.089815534000000002, 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.08996643, 2024-04-18T21:52:57.704 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.612043+0000", 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.612043+0000", 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:57.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.612043+0000", 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4900000000000002e-07 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.612055+0000", 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1513e-05 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:57.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.612070+0000", 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5099999999999999e-05 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.612083+0000", 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3407000000000001e-05 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.612194+0000", 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110888 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:57.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:57.612204+0000", 2024-04-18T21:52:57.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9459999999999993e-06 2024-04-18T21:52:57.708 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.708 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:57.708 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.708 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:57.708 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-04-18T21:52:57.708 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 2 2024-04-18T21:52:57.708 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:57.713 INFO:tasks.rados.rados.0.smithi097.stdout:332: done (7 left) 2024-04-18T21:52:57.713 INFO:tasks.rados.rados.0.smithi097.stdout:338: done (6 left) 2024-04-18T21:52:57.713 INFO:tasks.rados.rados.0.smithi097.stdout:339: copy_from oid 44 from oid 10 current snap is 24 2024-04-18T21:52:57.713 INFO:tasks.rados.rados.0.smithi097.stdout:340: read oid 35 snap -1 2024-04-18T21:52:57.713 INFO:tasks.rados.rados.0.smithi097.stdout:340: expect (ObjNum 118 snap 23 seq_num 118) 2024-04-18T21:52:57.713 INFO:tasks.rados.rados.0.smithi097.stdout:341: snap_create 2024-04-18T21:52:57.714 INFO:tasks.rados.rados.0.smithi097.stdout:331: finishing write tid 1 to smithi09731216-46 2024-04-18T21:52:57.714 INFO:tasks.rados.rados.0.smithi097.stdout:331: finishing write tid 2 to smithi09731216-46 2024-04-18T21:52:57.714 INFO:tasks.rados.rados.0.smithi097.stdout:331: finishing write tid 3 to smithi09731216-46 2024-04-18T21:52:57.714 INFO:tasks.rados.rados.0.smithi097.stdout:329: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-18T21:52:57.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:52:57.729 INFO:tasks.rados.rados.0.smithi097.stdout:331: finishing write tid 4 to smithi09731216-46 2024-04-18T21:52:57.730 INFO:tasks.rados.rados.0.smithi097.stdout:328: finishing write tid 3 to smithi09731216-38 2024-04-18T21:52:57.730 INFO:tasks.rados.rados.0.smithi097.stdout:328: finishing write tid 4 to smithi09731216-38 2024-04-18T21:52:57.730 INFO:tasks.rados.rados.0.smithi097.stdout:328: finishing write tid 5 to smithi09731216-38 2024-04-18T21:52:57.771 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:57.767+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:57.828 INFO:tasks.rados.rados.0.smithi097.stdout:328: finishing write tid 6 to smithi09731216-38 2024-04-18T21:52:57.828 INFO:tasks.rados.rados.0.smithi097.stdout:328: oid 38 updating version 0 to 197 2024-04-18T21:52:57.828 INFO:tasks.rados.rados.0.smithi097.stdout:328: oid 38 updating version 197 to 198 2024-04-18T21:52:57.829 INFO:tasks.rados.rados.0.smithi097.stdout:328: oid 38 version 198 is already newer than 195 2024-04-18T21:52:57.829 INFO:tasks.rados.rados.0.smithi097.stdout:328: oid 38 version 198 is already newer than 194 2024-04-18T21:52:57.829 INFO:tasks.rados.rados.0.smithi097.stdout:328: oid 38 version 198 is already newer than 196 2024-04-18T21:52:57.829 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 38 v 198 (ObjNum 130 snap 24 seq_num 130) dirty exists 2024-04-18T21:52:57.829 INFO:tasks.rados.rados.0.smithi097.stdout:328: left oid 38 (ObjNum 130 snap 24 seq_num 130) 2024-04-18T21:52:57.838 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:57.838 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:52:57.838 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-04-18T21:52:57.838 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-04-18T21:52:57.838 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:57.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:52:57.872 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:52:57.866+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:52:57.913 INFO:tasks.rados.rados.0.smithi097.stdout:335: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-18T21:52:57.918 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:52:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:52:57.922 INFO:tasks.rados.rados.0.smithi097.stdout:331: finishing write tid 5 to smithi09731216-46 2024-04-18T21:52:57.948 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:52:57.949 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:52:57.973 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:57.971+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:57.985 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:52:57.985 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:52:58.021 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:52:58.021 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:52:58.026 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:97 3.4s0 3:209f2930:::smithi09731216-33:head [append 0~540672,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.037501003000003, 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4026191889999999, 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.027 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985856+0000", 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.985858+0000", 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.92e-06 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987540+0000", 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001681463 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987566+0000", 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6043000000000001e-05 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987586+0000", 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9930000000000001e-05 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087950+0000", 2024-04-18T21:52:58.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100363824 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.087956+0000", 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9200000000000001e-06 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388374+0000", 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3004186340000001 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388475+0000", 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100455 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:98 3.4s0 3:209f2930:::smithi09731216-33:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.035739601000003, 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.401002842, 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.032 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 98 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987617+0000", 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987618+0000", 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5e-06 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987625+0000", 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6769999999999998e-06 2024-04-18T21:52:58.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987637+0000", 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1693e-05 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987646+0000", 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3700000000000001e-06 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088140+0000", 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10049390900000001 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088145+0000", 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3600000000000004e-06 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388530+0000", 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300384269 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388620+0000", 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0063999999999997e-05 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:99 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.035696861000005, 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.503241456, 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.037 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:58.038 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.038 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987660+0000", 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987661+0000", 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5999999999999991e-07 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987664+0000", 2024-04-18T21:52:58.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6459999999999999e-06 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987675+0000", 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0967e-05 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:46.987684+0000", 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6929999999999996e-06 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088241+0000", 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100557049 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:47.088244+0000", 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2729999999999998e-06 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388746+0000", 2024-04-18T21:52:58.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.300502061 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.388878+0000", 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131533 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:48.490901+0000", 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102023274 2024-04-18T21:52:58.043 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:218 3.bs0 3:d90895f4:::smithi09731216-9:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 65.007223506000003, 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61382267800000001, 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.044 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 218 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016133+0000", 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016134+0000", 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3e-06 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016141+0000", 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5719999999999999e-06 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016161+0000", 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9814e-05 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016177+0000", 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6028e-05 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016365+0000", 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018808400000000001 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.016384+0000", 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8899999999999999e-05 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026326+0000", 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0099418909999999996 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.026591+0000", 2024-04-18T21:52:58.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026506899999999999 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.027603+0000", 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010121799999999999 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.128223+0000", 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100619874 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629817+0000", 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50159395299999998 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:53.629956+0000", 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013901300000000001 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.050 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:300 3.bs0 3:d90895f4:::smithi09731216-9:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 59.958097187999996, 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.71857623199999998, 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 300 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065259+0000", 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065261+0000", 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9690000000000001e-06 2024-04-18T21:52:58.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065272+0000", 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0545e-05 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065299+0000", 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6927000000000002e-05 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065319+0000", 2024-04-18T21:52:58.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0035e-05 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165723+0000", 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100404099 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165747+0000", 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.438e-05 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.279995+0000", 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.114247892 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.280276+0000", 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028071299999999998 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281155+0000", 2024-04-18T21:52:58.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087942000000000001 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.381976+0000", 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100820639 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783683+0000", 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.40170656199999999 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783836+0000", 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000153051 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:301 3.bs0 3:d90895f4:::smithi09731216-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 59.957965033999997, 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.82307189599999997, 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.057 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 301 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065391+0000", 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065393+0000", 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.553e-06 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065401+0000", 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9719999999999992e-06 2024-04-18T21:52:58.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065425+0000", 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3547e-05 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.065447+0000", 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2234000000000001e-05 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165843+0000", 2024-04-18T21:52:58.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100395994 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.165860+0000", 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6871000000000002e-05 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281656+0000", 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11579645099999999 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:58.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.281793+0000", 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013725 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.783958+0000", 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50216431100000003 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.784112+0000", 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015411799999999999 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:51:58.888463+0000", 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10435159500000001 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:422 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~294912] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 43.576476581999998, 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61132010699999995, 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.063 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 422 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446880+0000", 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.446882+0000", 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5459999999999998e-06 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447318+0000", 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00043564599999999998 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447349+0000", 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0888e-05 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447365+0000", 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5593000000000001e-05 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447499+0000", 2024-04-18T21:52:58.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013438999999999999 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447509+0000", 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6509999999999996e-06 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.556315+0000", 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10880680299999999 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.556603+0000", 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028725400000000002 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.558288+0000", 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001684786 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638308+0000", 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.080020547999999997 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058105+0000", 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41979719399999998 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058200+0000", 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4808000000000004e-05 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.069 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:423 3.2s0 3:44bf1f18:::smithi09731216-13:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 43.575959439000002, 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61117605200000003, 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 423 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447397+0000", 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447398+0000", 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.353e-06 2024-04-18T21:52:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447416+0000", 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7353000000000001e-05 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447431+0000", 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5645e-05 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447441+0000", 2024-04-18T21:52:58.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2690000000000005e-06 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447569+0000", 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012837999999999999 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447573+0000", 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3270000000000002e-06 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.559335+0000", 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11176207000000001 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.559597+0000", 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026142 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.560692+0000", 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010952799999999999 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.638310+0000", 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.077617829999999999 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058483+0000", 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.42017316199999999 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.058573+0000", 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9963000000000006e-05 2024-04-18T21:52:58.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:424 3.2s0 3:44bf1f18:::smithi09731216-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 43.575861346000003, 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.81405277300000001, 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.076 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 424 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447495+0000", 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447496+0000", 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1699999999999997e-07 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447501+0000", 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5059999999999997e-06 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447513+0000", 2024-04-18T21:52:58.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2592000000000001e-05 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447526+0000", 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2309e-05 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.547672+0000", 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10014685800000001 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:58.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.547684+0000", 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1232e-05 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.756870+0000", 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.209186237 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.756965+0000", 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5280999999999998e-05 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.059024+0000", 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30205871899999998 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.059128+0000", 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010387600000000001 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.261548+0000", 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.202420246 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.081 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:435 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 43.575766547000001, 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.58911970899999999, 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.082 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 435 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447590+0000", 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447591+0000", 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2270000000000001e-06 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447598+0000", 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6630000000000004e-06 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447616+0000", 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8587999999999999e-05 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447631+0000", 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4326e-05 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447673+0000", 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2159000000000001e-05 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447677+0000", 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0799999999999999e-06 2024-04-18T21:52:58.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639051+0000", 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19137447299999999 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639173+0000", 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121443 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.036710+0000", 2024-04-18T21:52:58.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39753674999999999 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:436 3.4s0 3:209f2930:::smithi09731216-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 43.575704305000002, 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.59017562599999995, 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.087 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 436 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447652+0000", 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447653+0000", 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1540000000000001e-06 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447661+0000", 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7449999999999995e-06 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447679+0000", 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8263000000000001e-05 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447693+0000", 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3642000000000001e-05 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447729+0000", 2024-04-18T21:52:58.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6124000000000002e-05 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.447733+0000", 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9620000000000004e-06 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639451+0000", 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19171812099999999 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:14.639522+0000", 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0956000000000003e-05 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:15.037828+0000", 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.39830565899999998 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:748 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~507904 in=507904b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:58.092 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.065761+0000", 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.9575959469999997, 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.75874637300000003, 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 748 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.065761+0000", 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.065761+0000", 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.065761+0000", 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7899999999999999e-07 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066076+0000", 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031490799999999997 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066085+0000", 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6459999999999999e-06 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066089+0000", 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6360000000000002e-06 2024-04-18T21:52:58.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.112513+0000", 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046423352000000001 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.112893+0000", 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00037991899999999999 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.115390+0000", 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024979429999999999 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.217476+0000", 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102085156 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.824270+0000", 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.60679436399999997 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.824507+0000", 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023697 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:749 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~458752 in=458752b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.066099+0000", 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.9572574810000001, 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.75929558799999997, 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.098 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 749 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066099+0000", 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066099+0000", 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066099+0000", 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4299999999999998e-07 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066632+0000", 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00053204999999999995 2024-04-18T21:52:58.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066641+0000", 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3610000000000007e-06 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066647+0000", 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7869999999999998e-06 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.218362+0000", 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15171483999999999 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.218541+0000", 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000179848 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.220248+0000", 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00170695 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.226744+0000", 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064952409999999997 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.825178+0000", 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.59843464700000004 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.825395+0000", 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021642100000000001 2024-04-18T21:52:58.103 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:750 3.2s0 3:44bf1f18:::smithi09731216-13:head [append 0~81920 in=81920b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.066655+0000", 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.9567019060000002, 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.85866839800000005, 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 750 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066655+0000", 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066655+0000", 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066655+0000", 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5999999999999999e-07 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066709+0000", 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4398999999999999e-05 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066716+0000", 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7159999999999997e-06 2024-04-18T21:52:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066720+0000", 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.754e-06 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.221359+0000", 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15463885899999999 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.221636+0000", 2024-04-18T21:52:58.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027681199999999998 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.223309+0000", 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016729550000000001 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.227024+0000", 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003715005 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:58.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.925139+0000", 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.69811540999999999 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.925323+0000", 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018402800000000001 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:751 3.2s0 3:44bf1f18:::smithi09731216-13:head [setxattr _header (57) in=64b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:58.109 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.066726+0000", 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.956630584, 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.858954412, 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 751 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066726+0000", 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066726+0000", 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066726+0000", 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2899999999999999e-07 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066729+0000", 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1830000000000001e-06 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066732+0000", 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.546e-06 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066735+0000", 2024-04-18T21:52:58.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.869e-06 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.224227+0000", 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15749200199999999 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.224524+0000", 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000296569 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.225960+0000", 2024-04-18T21:52:58.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014364709999999999 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.232257+0000", 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006296683 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.925509+0000", 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.69325246200000001 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.925680+0000", 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017119800000000001 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.4509.0:752 3.2s0 3:44bf1f18:::smithi09731216-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.066739+0000", 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.9566176640000004, 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1179942899999999, 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.115 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.4509", 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 752 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066739+0000", 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066739+0000", 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066739+0000", 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.84e-07 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066741+0000", 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.412e-06 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066746+0000", 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5160000000000002e-06 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.066751+0000", 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6779999999999999e-06 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.226474+0000", 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.159723478 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for rw locks", 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.226628+0000", 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000154015 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.030872+0000", 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.80424367600000002 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-04-18T21:52:58.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.031036+0000", 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000164332 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.184733+0000", 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.153696899 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 94/84 ECSubWriteReply(tid=150, last_complete=94'32, committed=1, applied=1))", 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.323896+0000", 2024-04-18T21:52:58.120 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.6994606880000003, 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.60144881299999997, 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323896+0000", 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323896+0000", 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323902+0000", 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4330000000000001e-06 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323912+0000", 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7429999999999997e-06 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.323939+0000", 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7389000000000002e-05 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.424037+0000", 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100097357 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.924990+0000", 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.50095311399999998 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.925345+0000", 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035477700000000002 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s3 95/90 ECSubWrite(tid=26, reqid=client.4509.0:763, at_version=0'0, trim_to=0'0, roll_forward_to=94'23))", 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.616868+0000", 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.4064880750000004, 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61075907399999996, 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.124 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616868+0000", 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616868+0000", 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616870+0000", 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9599999999999999e-06 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616917+0000", 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6768999999999999e-05 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616960+0000", 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2354999999999997e-05 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.616986+0000", 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6605999999999999e-05 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.027965+0000", 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.41097915499999998 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.027987+0000", 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1325999999999999e-05 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.227564+0000", 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19957730200000001 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.227628+0000", 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3601000000000003e-05 2024-04-18T21:52:58.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s3 95/90 ECSubWrite(tid=27, reqid=client.4509.0:764, at_version=95'24, trim_to=0'0, roll_forward_to=94'23))", 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.618817+0000", 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.4045398479999998, 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.60887783500000003, 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:58.129 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.618817+0000", 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.618817+0000", 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.618818+0000", 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6700000000000001e-06 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.619162+0000", 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00034409000000000002 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.619202+0000", 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9959000000000001e-05 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.619228+0000", 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5125000000000001e-05 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.028351+0000", 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.409123087 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.028367+0000", 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6101999999999999e-05 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.227636+0000", 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.19926956400000001 2024-04-18T21:52:58.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.227695+0000", 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8238e-05 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDECSubOpWrite(3.12s3 95/90 ECSubWrite(tid=28, reqid=client.4509.0:765, at_version=95'25, trim_to=0'0, roll_forward_to=94'23))", 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-04-18T21:52:51.620495+0000", 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.4028611069999997, 2024-04-18T21:52:58.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.60726760199999996, 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.620495+0000", 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.620495+0000", 2024-04-18T21:52:58.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.620497+0000", 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2330000000000001e-06 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.620566+0000", 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9125e-05 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.620599+0000", 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2725999999999998e-05 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:51.720680+0000", 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100081375 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.029380+0000", 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.30870007199999999 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_started", 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.029398+0000", 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7716000000000001e-05 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_committed", 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.227701+0000", 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.198302957 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-04-18T21:52:52.227763+0000", 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2397999999999994e-05 2024-04-18T21:52:58.138 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.139 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.139 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.139 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-18T21:52:58.139 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-18T21:52:58.139 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:52:58.140 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:52:58.140 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:52:58.141 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:52:58.141 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:52:58.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:52:58.142 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:52:58.142 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:52:58.143 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:52:58.143 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:52:58.143 INFO:tasks.rados.rados.0.smithi097.stdout:334: expect (ObjNum 113 snap 23 seq_num 113) 2024-04-18T21:52:58.144 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:58.134+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:58.233 INFO:tasks.rados.rados.0.smithi097.stdout:336: finishing copy_from racing read to smithi09731216-16 2024-04-18T21:52:58.241 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:58.238+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:58.245 INFO:tasks.rados.rados.0.smithi097.stdout:340: expect (ObjNum 118 snap 23 seq_num 118) 2024-04-18T21:52:58.318 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:58.318 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:58.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:52:58.381 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:52:58.379+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:52:58.383 INFO:tasks.rados.rados.0.smithi097.stdout:331: finishing write tid 6 to smithi09731216-46 2024-04-18T21:52:58.383 INFO:tasks.rados.rados.0.smithi097.stdout:331: oid 46 updating version 0 to 41 2024-04-18T21:52:58.383 INFO:tasks.rados.rados.0.smithi097.stdout:331: oid 46 updating version 41 to 43 2024-04-18T21:52:58.383 INFO:tasks.rados.rados.0.smithi097.stdout:331: oid 46 version 43 is already newer than 42 2024-04-18T21:52:58.383 INFO:tasks.rados.rados.0.smithi097.stdout:331: oid 46 version 43 is already newer than 39 2024-04-18T21:52:58.383 INFO:tasks.rados.rados.0.smithi097.stdout:331: oid 46 version 43 is already newer than 40 2024-04-18T21:52:58.383 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 46 v 43 (ObjNum 131 snap 24 seq_num 131) dirty exists 2024-04-18T21:52:58.383 INFO:tasks.rados.rados.0.smithi097.stdout:331: left oid 46 (ObjNum 131 snap 24 seq_num 131) 2024-04-18T21:52:58.384 INFO:tasks.rados.rados.0.smithi097.stdout:339: finishing copy_from racing read to smithi09731216-44 2024-04-18T21:52:58.404 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:52:58.409 INFO:tasks.rados.rados.0.smithi097.stdout:328: done (8 left) 2024-04-18T21:52:58.409 INFO:tasks.rados.rados.0.smithi097.stdout:329: done (7 left) 2024-04-18T21:52:58.409 INFO:tasks.rados.rados.0.smithi097.stdout:331: done (6 left) 2024-04-18T21:52:58.409 INFO:tasks.rados.rados.0.smithi097.stdout:334: done (5 left) 2024-04-18T21:52:58.409 INFO:tasks.rados.rados.0.smithi097.stdout:335: done (4 left) 2024-04-18T21:52:58.409 INFO:tasks.rados.rados.0.smithi097.stdout:340: done (3 left) 2024-04-18T21:52:58.409 INFO:tasks.rados.rados.0.smithi097.stdout:341: done (2 left) 2024-04-18T21:52:58.409 INFO:tasks.rados.rados.0.smithi097.stdout:342: snap_create 2024-04-18T21:52:58.432 INFO:tasks.rados.rados.0.smithi097.stdout:336: finishing copy_from to smithi09731216-16 2024-04-18T21:52:58.432 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 16 v 34 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-04-18T21:52:58.442 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:58.439+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:58.459 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:58.460 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:58.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:52:58.530 INFO:tasks.rados.rados.0.smithi097.stdout:339: finishing copy_from to smithi09731216-44 2024-04-18T21:52:58.531 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 44 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-18T21:52:58.543 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:58.538+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:58.633 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:58.634 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:58.645 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:58.639+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:58.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:52:58.745 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:52:58.742+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:52:58.779 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:58.779 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:58.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:52:58.847 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:58.843+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:58.947 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:58.946+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:58.966 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:58.967 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:58.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:52:59.050 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:52:59.046+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:52:59.105 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:59.105 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:59.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:52:59.165 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:59.162+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:59.237 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:59.240 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:59.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:52:59.255 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:52:59.251+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:52:59.355 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:59.350+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:59.419 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:59.420 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:59.421 INFO:tasks.rados.rados.0.smithi097.stdout:336: done (2 left) 2024-04-18T21:52:59.421 INFO:tasks.rados.rados.0.smithi097.stdout:339: done (1 left) 2024-04-18T21:52:59.421 INFO:tasks.rados.rados.0.smithi097.stdout:342: done (0 left) 2024-04-18T21:52:59.421 INFO:tasks.rados.rados.0.smithi097.stdout:343: rollback oid 14 current snap is 26 2024-04-18T21:52:59.421 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 14 to 24 2024-04-18T21:52:59.421 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 9 current snap is 26 2024-04-18T21:52:59.421 INFO:tasks.rados.rados.0.smithi097.stdout:344: seq_num 134 ranges {2441216=704512,3145728=475136,3620864=573440,4194304=507904} 2024-04-18T21:52:59.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:52:59.448 INFO:tasks.rados.rados.0.smithi097.stdout:344: writing smithi09731216-9 from 2441216 to 3145728 tid 1 2024-04-18T21:52:59.453 INFO:tasks.rados.rados.0.smithi097.stdout:344: writing smithi09731216-9 from 3145728 to 3620864 tid 2 2024-04-18T21:52:59.457 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:52:59.450+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:52:59.458 INFO:tasks.rados.rados.0.smithi097.stdout:344: writing smithi09731216-9 from 3620864 to 4194304 tid 3 2024-04-18T21:52:59.463 INFO:tasks.rados.rados.0.smithi097.stdout:344: writing smithi09731216-9 from 4194304 to 4702208 tid 4 2024-04-18T21:52:59.463 INFO:tasks.rados.rados.0.smithi097.stdout:345: read oid 29 snap -1 2024-04-18T21:52:59.463 INFO:tasks.rados.rados.0.smithi097.stdout:345: expect (ObjNum 126 snap 24 seq_num 126) 2024-04-18T21:52:59.464 INFO:tasks.rados.rados.0.smithi097.stdout:346: rollback oid 35 current snap is 26 2024-04-18T21:52:59.464 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 35 to 25 2024-04-18T21:52:59.464 INFO:tasks.rados.rados.0.smithi097.stdout:347: snap_create 2024-04-18T21:52:59.557 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:59.555+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:59.573 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:59.573 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:59.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:52:59.658 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:52:59.655+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:52:59.759 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:59.759 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:59.759 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:52:59.755+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:52:59.773 INFO:tasks.rados.rados.0.smithi097.stdout:344: finishing write tid 1 to smithi09731216-9 2024-04-18T21:52:59.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:52:59.860 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:52:59.854+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:52:59.866 INFO:tasks.rados.rados.0.smithi097.stdout:346: finishing rollback tid 0 to smithi09731216-35 2024-04-18T21:52:59.867 INFO:tasks.rados.rados.0.smithi097.stdout:346: finishing rollback tid 1 to smithi09731216-35 2024-04-18T21:52:59.867 INFO:tasks.rados.rados.0.smithi097.stdout:346: finishing rollback tid 2 to smithi09731216-35 2024-04-18T21:52:59.867 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 35 v 47 (ObjNum 118 snap 23 seq_num 118) dirty exists 2024-04-18T21:52:59.875 INFO:tasks.rados.rados.0.smithi097.stdout:345: expect (ObjNum 126 snap 24 seq_num 126) 2024-04-18T21:52:59.896 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:52:59.896 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:52:59.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:52:59.961 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:52:59.958+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:52:59.971 INFO:tasks.rados.rados.0.smithi097.stdout:344: finishing write tid 2 to smithi09731216-9 2024-04-18T21:52:59.971 INFO:tasks.rados.rados.0.smithi097.stdout:344: finishing write tid 3 to smithi09731216-9 2024-04-18T21:52:59.971 INFO:tasks.rados.rados.0.smithi097.stdout:344: finishing write tid 4 to smithi09731216-9 2024-04-18T21:52:59.971 INFO:tasks.rados.rados.0.smithi097.stdout:344: finishing write tid 5 to smithi09731216-9 2024-04-18T21:53:00.025 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:00.025 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:00.034 INFO:tasks.rados.rados.0.smithi097.stdout:343: finishing rollback tid 0 to smithi09731216-14 2024-04-18T21:53:00.034 INFO:tasks.rados.rados.0.smithi097.stdout:343: finishing rollback tid 1 to smithi09731216-14 2024-04-18T21:53:00.035 INFO:tasks.rados.rados.0.smithi097.stdout:343: finishing rollback tid 2 to smithi09731216-14 2024-04-18T21:53:00.035 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 14 v 91 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-18T21:53:00.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:53:00.081 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:00.078+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:00.136 INFO:tasks.rados.rados.0.smithi097.stdout:344: finishing write tid 6 to smithi09731216-9 2024-04-18T21:53:00.136 INFO:tasks.rados.rados.0.smithi097.stdout:344: oid 9 updating version 0 to 63 2024-04-18T21:53:00.136 INFO:tasks.rados.rados.0.smithi097.stdout:344: oid 9 updating version 63 to 64 2024-04-18T21:53:00.136 INFO:tasks.rados.rados.0.smithi097.stdout:344: oid 9 version 64 is already newer than 62 2024-04-18T21:53:00.136 INFO:tasks.rados.rados.0.smithi097.stdout:344: oid 9 version 64 is already newer than 61 2024-04-18T21:53:00.136 INFO:tasks.rados.rados.0.smithi097.stdout:344: oid 9 version 64 is already newer than 60 2024-04-18T21:53:00.136 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 9 v 64 (ObjNum 134 snap 26 seq_num 134) dirty exists 2024-04-18T21:53:00.136 INFO:tasks.rados.rados.0.smithi097.stdout:344: left oid 9 (ObjNum 134 snap 26 seq_num 134) 2024-04-18T21:53:00.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:53:00.163 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:53:00.158+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:53:00.205 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:00.206 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:00.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:53:00.269 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:00.266+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:00.296 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:53:00.296 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:53:00.312 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:53:00.312 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:53:00.329 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:53:00.330 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:53:00.342 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:00.342 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:00.347 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:53:00.347 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:53:00.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:53:00.364 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:53:00.359+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:53:00.368 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:53:00.368 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:53:00.388 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:53:00.388 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:53:00.408 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:53:00.408 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:53:00.425 INFO:tasks.rados.rados.0.smithi097.stdout:343: done (4 left) 2024-04-18T21:53:00.425 INFO:tasks.rados.rados.0.smithi097.stdout:344: done (3 left) 2024-04-18T21:53:00.425 INFO:tasks.rados.rados.0.smithi097.stdout:345: done (2 left) 2024-04-18T21:53:00.425 INFO:tasks.rados.rados.0.smithi097.stdout:346: done (1 left) 2024-04-18T21:53:00.425 INFO:tasks.rados.rados.0.smithi097.stdout:347: done (0 left) 2024-04-18T21:53:00.425 INFO:tasks.rados.rados.0.smithi097.stdout:348: rollback oid 50 current snap is 27 2024-04-18T21:53:00.426 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 50 to 26 2024-04-18T21:53:00.426 INFO:tasks.rados.rados.0.smithi097.stdout:349: rmattr oid 38 current snap is 27 2024-04-18T21:53:00.426 INFO:tasks.rados.rados.0.smithi097.stdout:349: done (1 left) 2024-04-18T21:53:00.426 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 21 current snap is 27 2024-04-18T21:53:00.426 INFO:tasks.rados.rados.0.smithi097.stdout:350: seq_num 135 ranges {0=720896,720896=622592,1343488=688128,2031616=245760} 2024-04-18T21:53:00.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-18T21:53:00.429 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:53:00.430 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:53:00.430 INFO:tasks.rados.rados.0.smithi097.stdout:350: writing smithi09731216-21 from 0 to 720896 tid 1 2024-04-18T21:53:00.437 INFO:tasks.rados.rados.0.smithi097.stdout:350: writing smithi09731216-21 from 720896 to 1343488 tid 2 2024-04-18T21:53:00.442 INFO:tasks.rados.rados.0.smithi097.stdout:350: writing smithi09731216-21 from 1343488 to 2031616 tid 3 2024-04-18T21:53:00.445 INFO:tasks.rados.rados.0.smithi097.stdout:350: writing smithi09731216-21 from 2031616 to 2277376 tid 4 2024-04-18T21:53:00.445 INFO:tasks.rados.rados.0.smithi097.stdout:351: copy_from oid 37 from oid 14 current snap is 27 2024-04-18T21:53:00.445 INFO:tasks.rados.rados.0.smithi097.stdout:352: rmattr oid 34 current snap is 27 2024-04-18T21:53:00.445 INFO:tasks.rados.rados.0.smithi097.stdout:352: done (3 left) 2024-04-18T21:53:00.445 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 12 current snap is 27 2024-04-18T21:53:00.445 INFO:tasks.rados.rados.0.smithi097.stdout:353: seq_num 136 ranges {4784128=737280,5521408=770048,6291456=540672,6832128=245760} 2024-04-18T21:53:00.455 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:53:00.455 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:53:00.465 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:53:00.458+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:53:00.476 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:53:00.476 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:53:00.482 INFO:tasks.rados.rados.0.smithi097.stdout:353: writing smithi09731216-12 from 4784128 to 5521408 tid 1 2024-04-18T21:53:00.487 INFO:tasks.rados.rados.0.smithi097.stdout:353: writing smithi09731216-12 from 5521408 to 6291456 tid 2 2024-04-18T21:53:00.491 INFO:tasks.rados.rados.0.smithi097.stdout:353: writing smithi09731216-12 from 6291456 to 6832128 tid 3 2024-04-18T21:53:00.493 INFO:tasks.rados.rados.0.smithi097.stdout:353: writing smithi09731216-12 from 6832128 to 7077888 tid 4 2024-04-18T21:53:00.493 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 38 current snap is 27 2024-04-18T21:53:00.493 INFO:tasks.rados.rados.0.smithi097.stdout:354: seq_num 137 ranges {4947968=753664,5701632=786432,6488064=655360,7143424=114688} 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.8s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.728580289000007, 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51395767000000003, 2024-04-18T21:53:00.495 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764361+0000", 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.61e-06 2024-04-18T21:53:00.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764368+0000", 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1600000000000001e-06 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764389+0000", 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1073e-05 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764409+0000", 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0006000000000001e-05 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866089+0000", 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101679558 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866221+0000", 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132022 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278264+0000", 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41204366100000001 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278317+0000", 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2580000000000001e-05 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:103 3.18s0 3:1b279e7b:::smithi09731216-35:head [append 0~573440,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:53:00.500 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.511921825000002, 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6106588159999999, 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 103 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981021+0000", 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5470000000000001e-06 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981778+0000", 2024-04-18T21:53:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00075675099999999995 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981811+0000", 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3200000000000001e-05 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981832+0000", 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0703000000000002e-05 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.277997+0000", 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29616472700000002 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278003+0000", 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3269999999999997e-06 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591581+0000", 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.313578111 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591677+0000", 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5450000000000003e-05 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:104 3.18s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.511056971000002, 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714877378, 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:00.505 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981885+0000", 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0260000000000002e-06 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981895+0000", 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7969999999999995e-06 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981918+0000", 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3116e-05 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981931+0000", 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2850000000000001e-05 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278086+0000", 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29615502199999999 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278088+0000", 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6230000000000001e-06 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696611+0000", 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418522515 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696760+0000", 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149429 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.510992665000003, 2024-04-18T21:53:00.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9142408849999999, 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981948+0000", 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981952+0000", 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8739999999999997e-06 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981963+0000", 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.133e-05 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981971+0000", 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2900000000000002e-06 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278161+0000", 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29618945299999999 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:00.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278163+0000", 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.734e-06 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696819+0000", 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418656006 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.697058+0000", 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023910700000000001 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.896188+0000", 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19913024800000001 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:227 3.19s0 3:9941efaf:::smithi09731216-5:head [read 0~81920,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 67.479780384999998, 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50662614500000003, 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:00.515 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 227 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013159+0000", 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013160+0000", 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1299999999999999e-07 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013167+0000", 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9670000000000004e-06 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013187+0000", 2024-04-18T21:53:00.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9726000000000001e-05 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013202+0000", 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5483000000000001e-05 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013460+0000", 2024-04-18T21:53:00.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025794499999999998 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013468+0000", 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8460000000000008e-06 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.315837+0000", 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30236893999999997 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.316131+0000", 2024-04-18T21:53:00.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029429400000000001 2024-04-18T21:53:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.519785+0000", 2024-04-18T21:53:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20365413099999999 2024-04-18T21:53:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.520 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:318 3.19s0 3:9da3af1e:::smithi09731216-23:head [delete] snapc 5=[2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 62.300066843000003, 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60663671399999997, 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.521 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 318 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192874+0000", 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2699999999999998e-07 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192878+0000", 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0239999999999999e-06 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192886+0000", 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2549999999999997e-06 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192890+0000", 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.357e-06 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372031+0000", 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.179140625 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372092+0000", 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1457999999999993e-05 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372626+0000", 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053349599999999995 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.374211+0000", 2024-04-18T21:53:00.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001584752 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799295+0000", 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42508415500000002 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799510+0000", 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021466500000000001 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.526 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:329 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 61.26929157, 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60620628200000004, 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 329 2024-04-18T21:53:00.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6600000000000003e-07 2024-04-18T21:53:00.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223651+0000", 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5919999999999999e-06 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223655+0000", 2024-04-18T21:53:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9890000000000003e-06 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223658+0000", 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2600000000000001e-06 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514422+0000", 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29076426599999999 2024-04-18T21:53:00.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514584+0000", 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000161275 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625434+0000", 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.110850424 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625669+0000", 2024-04-18T21:53:00.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234315 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.829854+0000", 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20418589500000001 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:360 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~475136] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.128753420000002, 2024-04-18T21:53:00.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98249101000000005, 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 360 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:53:00.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364189+0000", 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8420000000000001e-06 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364486+0000", 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029724500000000002 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364517+0000", 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0994000000000002e-05 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364530+0000", 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2330999999999999e-05 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364644+0000", 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114673 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364657+0000", 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2258e-05 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.394826+0000", 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.030169702 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395495+0000", 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00066827599999999998 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.821805+0000", 2024-04-18T21:53:00.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42631051199999997 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.822025+0000", 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000220084 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824602+0000", 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025771510000000002 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944319+0000", 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11971646900000001 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346144+0000", 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401824764 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346677+0000", 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053370900000000005 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:361 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~622592 in=622592b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.124013189000003, 2024-04-18T21:53:00.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2802270259999999, 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 361 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368928+0000", 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6559999999999999e-06 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369217+0000", 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028880699999999997 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369238+0000", 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0914999999999999e-05 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369247+0000", 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1910000000000006e-06 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369333+0000", 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5482000000000006e-05 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:00.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369345+0000", 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2147000000000001e-05 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395635+0000", 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026290322000000001 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395761+0000", 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126348 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837828+0000", 2024-04-18T21:53:00.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44206649199999998 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837999+0000", 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017102900000000001 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.840298+0000", 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022987670000000002 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944332+0000", 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10403438199999999 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648919+0000", 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70458679599999996 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649154+0000", 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234692 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.547 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:362 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~147456 in=147456b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.123033563, 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.279510908, 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 362 2024-04-18T21:53:00.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369907+0000", 2024-04-18T21:53:00.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0529999999999999e-06 2024-04-18T21:53:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369994+0000", 2024-04-18T21:53:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6688999999999996e-05 2024-04-18T21:53:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370003+0000", 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3789999999999995e-06 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370010+0000", 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9229999999999996e-06 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370098+0000", 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8146000000000006e-05 2024-04-18T21:53:00.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370122+0000", 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3329e-05 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395809+0000", 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025687331000000001 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395945+0000", 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013612499999999999 2024-04-18T21:53:00.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941837+0000", 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54589167500000002 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941940+0000", 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103435 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942633+0000", 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069312599999999996 2024-04-18T21:53:00.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244973+0000", 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30233927599999999 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649234+0000", 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40426141100000001 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649417+0000", 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018301 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:363 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.122917389000001, 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.380372242, 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.555 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 363 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370023+0000", 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.75e-07 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370027+0000", 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8489999999999999e-06 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370039+0000", 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2013e-05 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370043+0000", 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6630000000000001e-06 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370169+0000", 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125506 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370180+0000", 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0798000000000001e-05 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396197+0000", 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026017723999999999 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396320+0000", 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122905 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943149+0000", 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54682909400000002 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943272+0000", 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012215699999999999 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943899+0000", 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000627106 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244975+0000", 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30107650600000002 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.749791+0000", 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50481555700000003 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.750395+0000", 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060378899999999998 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:365 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~491520] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.119307655999997, 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97351366699999997, 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 365 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373634+0000", 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.024e-06 2024-04-18T21:53:00.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373873+0000", 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000238459 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373899+0000", 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6461999999999998e-05 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373912+0000", 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2823000000000001e-05 2024-04-18T21:53:00.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374028+0000", 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115758 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374050+0000", 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2314e-05 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.601683+0000", 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22763349599999999 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.602117+0000", 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043313999999999999 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828190+0000", 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226073089 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828481+0000", 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029162099999999999 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.830297+0000", 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018160229999999999 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944323+0000", 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114025979 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346866+0000", 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40254325000000002 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.347146+0000", 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027922899999999998 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.569 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:366 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~770048] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.113688725000003, 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.067907554, 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.570 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 366 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379253+0000", 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9609999999999999e-06 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379714+0000", 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046141699999999999 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379730+0000", 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5585000000000001e-05 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379738+0000", 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379830+0000", 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2551000000000002e-05 2024-04-18T21:53:00.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379852+0000", 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.143e-05 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.604888+0000", 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225036559 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:00.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.605008+0000", 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119936 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831601+0000", 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226592508 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831778+0000", 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017783799999999999 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.834089+0000", 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023105930000000001 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944325+0000", 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11023567500000001 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.446970+0000", 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50264567000000004 2024-04-18T21:53:00.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.447159+0000", 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018811199999999999 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:367 3.11s0 3:8af06b78:::smithi09731216-10:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.113183808000002, 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1678633460000001, 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 367 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:53:00.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4899999999999995e-07 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379760+0000", 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4170000000000001e-06 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379766+0000", 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7459999999999997e-06 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379769+0000", 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.608e-06 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379929+0000", 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015980200000000001 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379953+0000", 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3764000000000002e-05 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707668+0000", 2024-04-18T21:53:00.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32771509500000001 2024-04-18T21:53:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707855+0000", 2024-04-18T21:53:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186987 2024-04-18T21:53:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835388+0000", 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127533496 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835835+0000", 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044705299999999999 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.836767+0000", 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00093163299999999996 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944330+0000", 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107563336 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547378+0000", 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60304798000000004 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547619+0000", 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024088000000000001 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:368 3.11s0 3:8af06b78:::smithi09731216-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.113147877999999, 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6760458580000002, 2024-04-18T21:53:00.585 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 368 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:53:00.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.91e-07 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379799+0000", 2024-04-18T21:53:00.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2690000000000002e-06 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379805+0000", 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7989999999999999e-06 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379809+0000", 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4929999999999998e-06 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379996+0000", 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186959 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.380015+0000", 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9837e-05 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707952+0000", 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32793653499999997 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.708113+0000", 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160536 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837592+0000", 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12947945499999999 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:00.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837760+0000", 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016796899999999999 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.647833+0000", 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.81007341499999996 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648038+0000", 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020446900000000001 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:05.055838+0000", 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.407799831 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:469 3.11s0 3:8fad7925:::smithi09731216-43:head [read 0~1966080,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.922949133000003, 2024-04-18T21:53:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50551296700000004, 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 469 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569991+0000", 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.569992+0000", 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.034e-06 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570000+0000", 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1680000000000004e-06 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570022+0000", 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2595999999999998e-05 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.570033+0000", 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0858e-05 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.571032+0000", 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00099923299999999998 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:18.571153+0000", 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120982 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.075504+0000", 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50435009600000003 2024-04-18T21:53:00.596 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:737 3.8s0 3:1b279e7b:::smithi09731216-35:head [append 0~147456] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:50.958634+0000", 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 9.5343057899999994, 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60400940000000003, 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.597 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 737 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958634+0000", 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958634+0000", 2024-04-18T21:53:00.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958635+0000", 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0070000000000001e-06 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958714+0000", 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9270999999999994e-05 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958725+0000", 2024-04-18T21:53:00.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0458999999999999e-05 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958732+0000", 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2559999999999996e-06 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.055021+0000", 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.096288755000000004 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.055129+0000", 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000108717 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.055852+0000", 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00072295599999999997 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.056907+0000", 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001055159 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.562580+0000", 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50567232699999998 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.562643+0000", 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3492999999999997e-05 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:738 3.8s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (57) in=7b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:50.958754+0000", 2024-04-18T21:53:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 9.5341853669999992, 2024-04-18T21:53:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60411800999999998, 2024-04-18T21:53:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.603 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 738 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958754+0000", 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958754+0000", 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958755+0000", 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.37e-07 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958758+0000", 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9679999999999998e-06 2024-04-18T21:53:00.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958763+0000", 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6620000000000002e-06 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.058839+0000", 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007514200000001 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.159007+0000", 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10016875 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.159108+0000", 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100278 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.159609+0000", 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00050155700000000002 2024-04-18T21:53:00.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.160564+0000", 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095452899999999997 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.562777+0000", 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40221319900000002 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.562872+0000", 2024-04-18T21:53:00.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5388000000000002e-05 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:813 3.9s0 3:9941efaf:::smithi09731216-5:head [copy-from ver 13] snapc 19=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:55.049609+0000", 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.4433304339999999, 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.53574875499999997, 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:00.609 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 813 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049609+0000", 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049609+0000", 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049612+0000", 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8159999999999998e-06 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049626+0000", 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3903000000000001e-05 2024-04-18T21:53:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049672+0000", 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6217000000000002e-05 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049690+0000", 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7479999999999999e-05 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.051766+0000", 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020764519999999999 2024-04-18T21:53:00.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.051939+0000", 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000173246 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.294069+0000", 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24212983299999999 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.361592+0000", 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.067522847999999996 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.584633+0000", 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22304118100000001 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.585358+0000", 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00072477900000000005 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.3s7 98/93 ECSubWrite(tid=349, reqid=osd.4.0:349, at_version=98'187, trim_to=0'0, roll_forward_to=98'185))", 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:55.193619+0000", 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.2993202799999999, 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.48718800200000001, 2024-04-18T21:53:00.615 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.193619+0000", 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.193619+0000", 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.193621+0000", 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1090000000000001e-06 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.193661+0000", 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0423999999999998e-05 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.193683+0000", 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2478999999999999e-05 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.293784+0000", 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10010021700000001 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.479314+0000", 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.185530415 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.479340+0000", 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6242999999999999e-05 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.680725+0000", 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.201384383 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.680807+0000", 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2731999999999993e-05 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.619 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.620 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.620 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:00.620 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:00.620 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:00.621 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:53:00.621 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:53:00.622 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:53:00.622 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:53:00.623 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:53:00.624 INFO:tasks.rados.rados.0.smithi097.stdout:354: writing smithi09731216-38 from 4947968 to 5701632 tid 1 2024-04-18T21:53:00.624 INFO:tasks.rados.rados.0.smithi097.stdout:354: writing smithi09731216-38 from 5701632 to 6488064 tid 2 2024-04-18T21:53:00.624 INFO:tasks.rados.rados.0.smithi097.stdout:354: writing smithi09731216-38 from 6488064 to 7143424 tid 3 2024-04-18T21:53:00.624 INFO:tasks.rados.rados.0.smithi097.stdout:354: writing smithi09731216-38 from 7143424 to 7258112 tid 4 2024-04-18T21:53:00.624 INFO:tasks.rados.rados.0.smithi097.stdout:355: copy_from oid 44 from oid 8 current snap is 27 2024-04-18T21:53:00.624 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 49 current snap is 27 2024-04-18T21:53:00.624 INFO:tasks.rados.rados.0.smithi097.stdout:356: seq_num 138 ranges {2048000=770048,2818048=540672,3358720=753664,4112384=262144} 2024-04-18T21:53:00.626 INFO:tasks.rados.rados.0.smithi097.stdout:356: writing smithi09731216-49 from 2048000 to 2818048 tid 1 2024-04-18T21:53:00.626 INFO:tasks.rados.rados.0.smithi097.stdout:356: writing smithi09731216-49 from 2818048 to 3358720 tid 2 2024-04-18T21:53:00.626 INFO:tasks.rados.rados.0.smithi097.stdout:356: writing smithi09731216-49 from 3358720 to 4112384 tid 3 2024-04-18T21:53:00.626 INFO:tasks.rados.rados.0.smithi097.stdout:356: writing smithi09731216-49 from 4112384 to 4374528 tid 4 2024-04-18T21:53:00.626 INFO:tasks.rados.rados.0.smithi097.stdout:357: snap_remove snap 24 2024-04-18T21:53:00.627 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:53:00.618+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:53:00.722 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:00.718+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:00.764 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:00.764 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:00.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.764 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 104/98 ECSubWrite(tid=234, reqid=client.4509.0:865, at_version=104'44, trim_to=0'0, roll_forward_to=100'42))", 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.594795+0000", 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.16721285899999999, 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16725459100000001, 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.594795+0000", 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.594795+0000", 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.594795+0000", 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3e-07 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.594838+0000", 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2561999999999997e-05 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.594863+0000", 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5256999999999999e-05 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.694949+0000", 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100085962 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695135+0000", 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018562 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695166+0000", 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0905000000000003e-05 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=370, last_complete=104'201, committed=1, applied=1))", 2024-04-18T21:53:00.768 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.603305+0000", 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.158702279, 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15885090599999999, 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.603305+0000", 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.603305+0000", 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.603306+0000", 2024-04-18T21:53:00.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3399999999999998e-07 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.603312+0000", 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4960000000000004e-06 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.603328+0000", 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6135999999999999e-05 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703415+0000", 2024-04-18T21:53:00.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100087523 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=371, last_complete=104'202, committed=1, applied=1))", 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.650627+0000", 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.111380771, 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.111605843, 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650627+0000", 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650627+0000", 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650627+0000", 2024-04-18T21:53:00.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1100000000000002e-07 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650630+0000", 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5129999999999999e-06 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650635+0000", 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3539999999999999e-06 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650639+0000", 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0010000000000003e-06 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=372, last_complete=104'203, committed=1, applied=1))", 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.650649+0000", 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.111358586, 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.111657796, 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.775 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650649+0000", 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650649+0000", 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650649+0000", 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4999999999999999e-07 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650651+0000", 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0260000000000002e-06 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650656+0000", 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7840000000000003e-06 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.650660+0000", 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8279999999999996e-06 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=371, last_complete=104'202, committed=1, applied=1))", 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.654266+0000", 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.10774203, 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.108113895, 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.778 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.654266+0000", 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.654266+0000", 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.654267+0000", 2024-04-18T21:53:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.367e-06 2024-04-18T21:53:00.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.654273+0000", 2024-04-18T21:53:00.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7660000000000004e-06 2024-04-18T21:53:00.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.654289+0000", 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6497000000000001e-05 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.654298+0000", 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3299999999999999e-06 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=372, last_complete=104'203, committed=1, applied=1))", 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.656804+0000", 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.105203727, 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105649724, 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.656804+0000", 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.656804+0000", 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.656805+0000", 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0920000000000001e-06 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.656810+0000", 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1730000000000003e-06 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.656822+0000", 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1419e-05 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.656828+0000", 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.63e-06 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.784 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 104/98 ECSubWrite(tid=235, reqid=client.4509.0:866, at_version=104'45, trim_to=0'0, roll_forward_to=100'42))", 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.694985+0000", 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.067022737999999998, 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.067545037000000002, 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.694985+0000", 2024-04-18T21:53:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.694985+0000", 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.694986+0000", 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3200000000000005e-07 2024-04-18T21:53:00.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695018+0000", 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2477000000000001e-05 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695031+0000", 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2492e-05 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695040+0000", 2024-04-18T21:53:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5009999999999993e-06 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.697666+0000", 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026262519999999999 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.697696+0000", 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9539e-05 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 104/98 ECSubWrite(tid=236, reqid=client.4509.0:867, at_version=104'46, trim_to=0'0, roll_forward_to=100'42))", 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.695090+0000", 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.066917862999999994, 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.06753286, 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695090+0000", 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695090+0000", 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695090+0000", 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4799999999999999e-07 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695116+0000", 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5610999999999999e-05 2024-04-18T21:53:00.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695125+0000", 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1179999999999998e-06 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695133+0000", 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8120000000000008e-06 2024-04-18T21:53:00.791 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=369, last_complete=104'200, committed=1, applied=1))", 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.695258+0000", 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.066749844000000003, 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.067437936000000004, 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.792 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695258+0000", 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695258+0000", 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695258+0000", 2024-04-18T21:53:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6199999999999998e-07 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695262+0000", 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4860000000000002e-06 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695271+0000", 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9870000000000005e-06 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695279+0000", 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2600000000000005e-06 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=370, last_complete=104'201, committed=1, applied=1))", 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.695289+0000", 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.066718747999999994, 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.067481844999999999, 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.795 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695289+0000", 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695289+0000", 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695289+0000", 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0100000000000001e-07 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695292+0000", 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5069999999999999e-06 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695299+0000", 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.17e-06 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695304+0000", 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1569999999999999e-06 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=371, last_complete=104'202, committed=1, applied=1))", 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.695311+0000", 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.066697115000000001, 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.067532884000000001, 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.798 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695311+0000", 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695311+0000", 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695311+0000", 2024-04-18T21:53:00.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9399999999999999e-07 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695313+0000", 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0090000000000002e-06 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695319+0000", 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.75e-06 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.695324+0000", 2024-04-18T21:53:00.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1019999999999996e-06 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=372, last_complete=104'203, committed=1, applied=1))", 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.701283+0000", 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.060724831999999999, 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.061633309999999997, 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.801 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.701283+0000", 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.701283+0000", 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.701284+0000", 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.499e-06 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.701290+0000", 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8819999999999998e-06 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.701304+0000", 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3814e-05 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.701325+0000", 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1246999999999998e-05 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:882 3.3s0 3:c26b2dda:::smithi09731216-38:head [append 0~753664 in=753664b] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.526340+0000", 2024-04-18T21:53:00.804 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.23566725099999999, 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23666235299999999, 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 882 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.526340+0000", 2024-04-18T21:53:00.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.526340+0000", 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.526343+0000", 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8059999999999999e-06 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.527541+0000", 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011978290000000001 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.527575+0000", 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3618999999999998e-05 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.527598+0000", 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3139999999999999e-05 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.532736+0000", 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051384830000000001 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.533074+0000", 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033765999999999999 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.539469+0000", 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063953910000000003 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.548640+0000", 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00917106 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=371, last_complete=104'202, committed=1, applied=1))", 2024-04-18T21:53:00.809 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.703462+0000", 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.058545561000000003, 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.059665334, 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703462+0000", 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703462+0000", 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703463+0000", 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.283e-06 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703475+0000", 2024-04-18T21:53:00.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1677e-05 2024-04-18T21:53:00.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703497+0000", 2024-04-18T21:53:00.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.16e-05 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703515+0000", 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8386e-05 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:883 3.3s0 3:c26b2dda:::smithi09731216-38:head [append 0~786432 in=786432b] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.531082+0000", 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.230925566, 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.232125478, 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.813 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 883 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.531082+0000", 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.531082+0000", 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.531085+0000", 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1e-06 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.532537+0000", 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014519870000000001 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.532563+0000", 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6101999999999998e-05 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.532579+0000", 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5642999999999999e-05 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.543537+0000", 2024-04-18T21:53:00.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010957972999999999 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.543822+0000", 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028460399999999999 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.546833+0000", 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030114949999999999 2024-04-18T21:53:00.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.555614+0000", 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0087806710000000003 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=372, last_complete=104'203, committed=1, applied=1))", 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.703553+0000", 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.058454889000000003, 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.059773171999999999, 2024-04-18T21:53:00.818 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703553+0000", 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703553+0000", 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703554+0000", 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7599999999999996e-07 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703561+0000", 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2899999999999997e-06 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703577+0000", 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6473e-05 2024-04-18T21:53:00.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.703592+0000", 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4752999999999999e-05 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:884 3.3s0 3:c26b2dda:::smithi09731216-38:head [append 0~655360 in=655360b] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.534430+0000", 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.22757761900000001, 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.228976443, 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.821 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 884 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.534430+0000", 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.534430+0000", 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.534433+0000", 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.83e-06 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.535890+0000", 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014574060000000001 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.535923+0000", 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2947000000000002e-05 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.535940+0000", 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6362e-05 2024-04-18T21:53:00.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.549069+0000", 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013129564999999999 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.549444+0000", 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000375282 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.552050+0000", 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026051329999999999 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.728233+0000", 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1761839 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 104/93 ECSubWriteReply(tid=369, last_complete=104'200, committed=1, applied=1))", 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.741351+0000", 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.020657035000000001, 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022179095999999999, 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.741351+0000", 2024-04-18T21:53:00.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.741351+0000", 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.741353+0000", 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9199999999999998e-06 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.741362+0000", 2024-04-18T21:53:00.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2459999999999992e-06 2024-04-18T21:53:00.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.741392+0000", 2024-04-18T21:53:00.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9926000000000001e-05 2024-04-18T21:53:00.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.741413+0000", 2024-04-18T21:53:00.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0893000000000001e-05 2024-04-18T21:53:00.830 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.830 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.830 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.831 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:885 3.3s0 3:c26b2dda:::smithi09731216-38:head [append 0~114688 in=114688b] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:00.831 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.536000+0000", 2024-04-18T21:53:00.831 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.226007811, 2024-04-18T21:53:00.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.227608805, 2024-04-18T21:53:00.832 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.832 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.832 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:00.832 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:00.832 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:00.832 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.832 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 885 2024-04-18T21:53:00.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.833 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536000+0000", 2024-04-18T21:53:00.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536000+0000", 2024-04-18T21:53:00.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536001+0000", 2024-04-18T21:53:00.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.59e-06 2024-04-18T21:53:00.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536217+0000", 2024-04-18T21:53:00.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021535200000000001 2024-04-18T21:53:00.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536245+0000", 2024-04-18T21:53:00.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8623e-05 2024-04-18T21:53:00.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536258+0000", 2024-04-18T21:53:00.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2703e-05 2024-04-18T21:53:00.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:00.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.553436+0000", 2024-04-18T21:53:00.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017177844000000001 2024-04-18T21:53:00.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:00.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.553649+0000", 2024-04-18T21:53:00.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021331200000000001 2024-04-18T21:53:00.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:00.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.554873+0000", 2024-04-18T21:53:00.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001224125 2024-04-18T21:53:00.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:00.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.728387+0000", 2024-04-18T21:53:00.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.17351409500000001 2024-04-18T21:53:00.841 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.841 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.841 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.842 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:886 3.3s0 3.5bb4d643 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:00.842 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.536279+0000", 2024-04-18T21:53:00.842 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.22572915099999999, 2024-04-18T21:53:00.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.227448025, 2024-04-18T21:53:00.842 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.842 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.843 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.843 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:00.843 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:00.843 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.843 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 886 2024-04-18T21:53:00.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.844 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536279+0000", 2024-04-18T21:53:00.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536279+0000", 2024-04-18T21:53:00.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536280+0000", 2024-04-18T21:53:00.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.15e-06 2024-04-18T21:53:00.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536286+0000", 2024-04-18T21:53:00.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1700000000000002e-06 2024-04-18T21:53:00.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536300+0000", 2024-04-18T21:53:00.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4574e-05 2024-04-18T21:53:00.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536314+0000", 2024-04-18T21:53:00.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3879e-05 2024-04-18T21:53:00.848 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.848 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.849 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.849 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:887 3.3s0 3.5bb4d643 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:00.849 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.536332+0000", 2024-04-18T21:53:00.849 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.22567526199999999, 2024-04-18T21:53:00.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.227477235, 2024-04-18T21:53:00.850 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:00.850 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:00.850 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:00.850 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:00.850 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:00.850 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:00.850 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 887 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536332+0000", 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536332+0000", 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:00.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536334+0000", 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3239999999999999e-06 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536339+0000", 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7259999999999999e-06 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:00.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536363+0000", 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.357e-05 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536379+0000", 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6288999999999999e-05 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 21 2024-04-18T21:53:00.853 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:00.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:53:00.855 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:53:00.850+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:53:00.956 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:00.950+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:00.983 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:00.983 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:00.983 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:53:00.983 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:53:00.983 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:00.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:53:01.056 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:01.050+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:01.122 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:01.122 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.865826236000004, 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82320560300000001, 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.123 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254837+0000", 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9000000000000002e-06 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255465+0000", 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000628435 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255498+0000", 2024-04-18T21:53:01.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2305999999999999e-05 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255523+0000", 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5137e-05 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078012+0000", 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82248975599999996 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078039+0000", 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7069000000000001e-05 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.865099330999996, 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82259326700000002, 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.127 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255563+0000", 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.72e-06 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255574+0000", 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0993e-05 2024-04-18T21:53:01.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255598+0000", 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4329999999999999e-05 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255615+0000", 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7569000000000001e-05 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078131+0000", 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82251584300000002 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078154+0000", 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2812000000000001e-05 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:93 3.3s0 3.fef99513 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:53:01.131 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.865011389000003, 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822627518, 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 93 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255650+0000", 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2330000000000001e-06 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255657+0000", 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5000000000000002e-06 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255681+0000", 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3354e-05 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255705+0000", 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3932999999999999e-05 2024-04-18T21:53:01.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078240+0000", 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822535609 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078276+0000", 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5889000000000001e-05 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:73 3.13s0 3:c8da3643:::smithi09731216-25:head [append 0~409600,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 73.923927918999993, 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.609707234, 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.136 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196734+0000", 2024-04-18T21:53:01.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829999999999999e-06 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196977+0000", 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024357200000000001 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196998+0000", 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0503e-05 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197013+0000", 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5536000000000002e-05 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197060+0000", 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6519e-05 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197061+0000", 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2900000000000002e-07 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591261+0000", 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3942002449999999 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591644+0000", 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038315000000000002 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593905+0000", 2024-04-18T21:53:01.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022609050000000001 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697736+0000", 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10383110700000001 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806236+0000", 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10850008999999999 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806439+0000", 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020309500000000001 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:74 3.13s0 3:c8da3643:::smithi09731216-25:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 73.923623574999993, 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7102205860000002, 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 74 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:53:01.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197038+0000", 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9000000000000005e-07 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197043+0000", 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197053+0000", 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0267e-05 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197066+0000", 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.327e-05 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197104+0000", 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8039e-05 2024-04-18T21:53:01.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197105+0000", 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8400000000000004e-07 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595282+0000", 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3981771869999999 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595543+0000", 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026070899999999997 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.596517+0000", 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00097469500000000003 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697956+0000", 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10143811899999999 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907075+0000", 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20911996799999999 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907257+0000", 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018164799999999999 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.13s0 3:c8a99f7f:::smithi09731216-31:head [append 0~507904,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 73.923097511999998, 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.812483013, 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.149 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8599999999999998e-07 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197879+0000", 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000315495 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197893+0000", 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4143e-05 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197908+0000", 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4932999999999999e-05 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197987+0000", 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9288000000000004e-05 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197989+0000", 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.137e-06 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.698727+0000", 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.500737242 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.699015+0000", 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028849499999999999 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.700958+0000", 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019429969999999999 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798717+0000", 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097759288999999999 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.009932+0000", 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.211214651 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010046+0000", 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113657 2024-04-18T21:53:01.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.13s0 3:c8a99f7f:::smithi09731216-31:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 73.922735670999998, 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8124102710000001, 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:53:01.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197925+0000", 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000005e-07 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197931+0000", 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6799999999999998e-06 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197943+0000", 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1620000000000001e-05 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197953+0000", 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-05 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7586000000000001e-05 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5599999999999995e-07 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702431+0000", 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.504419629 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702670+0000", 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023948700000000001 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.703977+0000", 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00130674 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798719+0000", 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094742397000000006 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010219+0000", 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21149920999999999 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010335+0000", 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011602 2024-04-18T21:53:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:234 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.086843543000001, 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.87684132699999995, 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 234 2024-04-18T21:53:01.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9599999999999998e-07 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033821+0000", 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.653e-06 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033834+0000", 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3303e-05 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033847+0000", 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2327000000000001e-05 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808105+0000", 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.77425798199999996 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808280+0000", 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017508500000000001 2024-04-18T21:53:01.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.910658+0000", 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102378081 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:307 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 63.034277181, 2024-04-18T21:53:01.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81219510800000005, 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 307 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086384+0000", 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9699999999999995e-07 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086388+0000", 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3900000000000003e-06 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086398+0000", 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8369999999999992e-06 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086406+0000", 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3599999999999996e-06 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086727+0000", 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032101900000000002 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086748+0000", 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0917e-05 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292674+0000", 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20592602600000001 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292826+0000", 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015169700000000001 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.898578+0000", 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60575206500000001 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.170 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:325 3.3s0 3:c26b2dda:::smithi09731216-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 61.894486295, 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.79501891000000002, 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.171 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 325 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4400000000000001e-07 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226176+0000", 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1100000000000001e-06 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226182+0000", 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5230000000000003e-06 2024-04-18T21:53:01.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226186+0000", 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0180000000000003e-06 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.230252+0000", 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040660369999999998 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.230395+0000", 2024-04-18T21:53:01.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014306400000000001 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.817971+0000", 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58757587300000003 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.818220+0000", 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024919800000000001 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:00.021193+0000", 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20297274300000001 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:432 3.3s0 3:c170a397:::smithi09731216-3:head [copy-from ver 11] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.674948442000002, 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.728029657, 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.176 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 432 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445712+0000", 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445713+0000", 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4759999999999999e-06 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445723+0000", 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.027e-05 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445745+0000", 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.126e-05 2024-04-18T21:53:01.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445770+0000", 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5803e-05 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445905+0000", 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013424800000000001 2024-04-18T21:53:01.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445914+0000", 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9230000000000008e-06 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.552790+0000", 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106876007 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.553298+0000", 2024-04-18T21:53:01.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00050816200000000005 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.967658+0000", 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41436041099999998 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.072477+0000", 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104818595 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173605+0000", 2024-04-18T21:53:01.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101128601 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.173741+0000", 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013590100000000001 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:533 3.3s0 3:c2fd059f:::smithi09731216-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.610394741999997, 2024-04-18T21:53:01.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.85709232000000002, 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 533 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510266+0000", 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-06 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510271+0000", 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7670000000000003e-06 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510283+0000", 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1965e-05 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.610415+0000", 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100132151 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.927765+0000", 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31734968699999999 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.927918+0000", 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000152702 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133621+0000", 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.205703102 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133704+0000", 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3040000000000005e-05 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.367358+0000", 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23365385799999999 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:608 3.1s1 3:809f338e:::smithi09731216-11:head [copy-from ver 112] snapc 14=[10,a,6] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e78)", 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.741532459999998, 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.881166288, 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 608 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379131+0000", 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1e-06 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379144+0000", 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2904e-05 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379188+0000", 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3909000000000003e-05 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379214+0000", 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6610000000000001e-05 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.675749+0000", 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29653525600000002 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.675755+0000", 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0129999999999999e-06 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.901288+0000", 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.225532912 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.901574+0000", 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028603400000000001 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.146507+0000", 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.244932181 2024-04-18T21:53:01.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.150976+0000", 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044695780000000001 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.259562+0000", 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10858633299999999 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.260294+0000", 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00073145800000000004 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:609 3.1s1 3:809f338e:::smithi09731216-11:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.741406100999999, 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7716583269999999, 2024-04-18T21:53:01.194 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 609 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379255+0000", 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.474e-06 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379263+0000", 2024-04-18T21:53:01.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7530000000000001e-06 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379287+0000", 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3416000000000001e-05 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379305+0000", 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.819e-05 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.775808+0000", 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39650268500000002 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.775813+0000", 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8329999999999999e-06 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.902311+0000", 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.126497995 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.902394+0000", 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2752999999999999e-05 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.149564+0000", 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.247170267 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.150153+0000", 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0005886 2024-04-18T21:53:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.150912+0000", 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00075936099999999996 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:680 3.3s0 3:c0b736af:::smithi09731216-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:45.560822+0000", 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.559838016, 2024-04-18T21:53:01.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.83707749099999995, 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 680 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560822+0000", 2024-04-18T21:53:01.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560822+0000", 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560823+0000", 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8700000000000004e-07 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560827+0000", 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7359999999999999e-06 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560837+0000", 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0349999999999999e-05 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560846+0000", 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5099999999999998e-06 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.591970+0000", 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031124037 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.592184+0000", 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021478400000000001 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.177742+0000", 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58555721100000002 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.178151+0000", 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000409178 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.397900+0000", 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.219748798 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:727 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~0] snapc 18=[17,10] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:49.474328+0000", 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.64633248, 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1869300279999999, 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 727 2024-04-18T21:53:01.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474328+0000", 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474328+0000", 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474331+0000", 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.943e-06 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474344+0000", 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.31e-05 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474376+0000", 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.222e-05 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474401+0000", 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4902999999999999e-05 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167528+0000", 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69312696200000001 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167532+0000", 2024-04-18T21:53:01.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5329999999999996e-06 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.111274+0000", 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94374193200000001 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.111739+0000", 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046505500000000003 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.113892+0000", 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021522770000000002 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.321845+0000", 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.207953264 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.661063+0000", 2024-04-18T21:53:01.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.33921768099999999 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.661258+0000", 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019515799999999999 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:728 3.3s0 3:c2fd059f:::smithi09731216-20:head [rollback 17] snapc 18=[17,10] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:49.474489+0000", 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.646171098, 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3914702019999998, 2024-04-18T21:53:01.212 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 728 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474489+0000", 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474489+0000", 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474490+0000", 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.063e-06 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474496+0000", 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9370000000000001e-06 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474526+0000", 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0389e-05 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474542+0000", 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5183e-05 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167563+0000", 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.693021675 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167564+0000", 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6000000000000003e-07 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.218010+0000", 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0504465030000001 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.218206+0000", 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000196021 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.219222+0000", 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010155660000000001 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.322206+0000", 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10298433999999999 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.865666+0000", 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.54345940500000001 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.865959+0000", 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029346 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.218 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:729 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~0] snapc 18=[17,10] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:49.474558+0000", 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.646101699000001, 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.392365496, 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 729 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474558+0000", 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474558+0000", 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474559+0000", 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9299999999999997e-07 2024-04-18T21:53:01.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474563+0000", 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6899999999999998e-06 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474574+0000", 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.08e-05 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474582+0000", 2024-04-18T21:53:01.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4100000000000008e-06 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167607+0000", 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69302461500000001 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167609+0000", 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5739999999999998e-06 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.325225+0000", 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.157615944 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.325417+0000", 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019197800000000001 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.326488+0000", 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010710439999999999 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.328451+0000", 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019627529999999998 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.866764+0000", 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.53831266099999997 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.866924+0000", 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016023400000000001 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:771 3.3s0 3:c170a397:::smithi09731216-3:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.055730+0000", 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.064929988999999, 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81168786100000001, 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 771 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055730+0000", 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055730+0000", 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055734+0000", 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4800000000000001e-06 2024-04-18T21:53:01.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055748+0000", 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.413e-05 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055779+0000", 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0854e-05 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055801+0000", 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2656000000000001e-05 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.326954+0000", 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.27115272000000001 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.327187+0000", 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023294099999999999 2024-04-18T21:53:01.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.867418+0000", 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.54023107999999997 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:821 3.3s0 3:c26b2dda:::smithi09731216-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:57.067499+0000", 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.0531608270000001, 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.760915016, 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 821 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067499+0000", 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067499+0000", 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067500+0000", 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7400000000000002e-07 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067504+0000", 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4060000000000001e-06 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067517+0000", 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3356000000000001e-05 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.167607+0000", 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100089574 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.176136+0000", 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0085291789999999996 2024-04-18T21:53:01.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.176279+0000", 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143793 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.722494+0000", 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.54621423999999996 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.722784+0000", 2024-04-18T21:53:01.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029031700000000001 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.828414+0000", 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10563027699999999 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.234 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:01.236 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:53:01.236 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:01.230+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:01.336 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:53:01.334+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:53:01.407 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:01.407 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:01.407 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:53:01.407 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:01.421 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:53:01.430 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is unset 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:357: done (7 left) 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:358: copy_from oid 20 from oid 29 current snap is 27 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:355: finishing copy_from to smithi09731216-44 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:355: got expected ENOENT (src dne) 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:355: finishing copy_from racing read to smithi09731216-44 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:350: finishing write tid 1 to smithi09731216-21 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:350: finishing write tid 2 to smithi09731216-21 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:350: finishing write tid 3 to smithi09731216-21 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:351: finishing copy_from racing read to smithi09731216-37 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:353: finishing write tid 1 to smithi09731216-12 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:353: finishing write tid 2 to smithi09731216-12 2024-04-18T21:53:01.438 INFO:tasks.rados.rados.0.smithi097.stdout:353: finishing write tid 3 to smithi09731216-12 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:348: finishing rollback tid 0 to smithi09731216-50 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:348: finishing rollback tid 1 to smithi09731216-50 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:348: finishing rollback tid 2 to smithi09731216-50 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 50 v 46 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:350: finishing write tid 4 to smithi09731216-21 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:356: finishing write tid 1 to smithi09731216-49 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:356: finishing write tid 2 to smithi09731216-49 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:356: finishing write tid 3 to smithi09731216-49 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:354: finishing write tid 1 to smithi09731216-38 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:354: finishing write tid 2 to smithi09731216-38 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:354: finishing write tid 3 to smithi09731216-38 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:354: finishing write tid 4 to smithi09731216-38 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:350: finishing write tid 5 to smithi09731216-21 2024-04-18T21:53:01.439 INFO:tasks.rados.rados.0.smithi097.stdout:356: finishing write tid 4 to smithi09731216-49 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:354: finishing write tid 5 to smithi09731216-38 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:351: finishing copy_from to smithi09731216-37 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 37 v 91 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:356: finishing write tid 5 to smithi09731216-49 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:350: finishing write tid 6 to smithi09731216-21 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:350: oid 21 updating version 0 to 60 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:350: oid 21 updating version 60 to 61 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:350: oid 21 updating version 61 to 62 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:350: oid 21 updating version 62 to 63 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:350: oid 21 version 63 is already newer than 59 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 21 v 63 (ObjNum 135 snap 27 seq_num 135) dirty exists 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:350: left oid 21 (ObjNum 135 snap 27 seq_num 135) 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:354: finishing write tid 6 to smithi09731216-38 2024-04-18T21:53:01.440 INFO:tasks.rados.rados.0.smithi097.stdout:354: oid 38 updating version 0 to 203 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:354: oid 38 version 203 is already newer than 202 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:354: oid 38 updating version 203 to 204 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:354: oid 38 version 204 is already newer than 200 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:354: oid 38 version 204 is already newer than 201 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 38 v 204 (ObjNum 137 snap 27 seq_num 137) dirty exists 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:354: left oid 38 (ObjNum 137 snap 27 seq_num 137) 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:348: done (7 left) 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:350: done (6 left) 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:351: done (5 left) 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:354: done (4 left) 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:355: done (3 left) 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:359: copy_from oid 37 from oid 1 current snap is 27 2024-04-18T21:53:01.441 INFO:tasks.rados.rados.0.smithi097.stdout:360: snap_create 2024-04-18T21:53:01.446 INFO:tasks.rados.rados.0.smithi097.stdout:353: finishing write tid 4 to smithi09731216-12 2024-04-18T21:53:01.446 INFO:tasks.rados.rados.0.smithi097.stdout:353: finishing write tid 5 to smithi09731216-12 2024-04-18T21:53:01.469 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:53:01.466+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:53:01.538 INFO:tasks.rados.rados.0.smithi097.stdout:356: finishing write tid 6 to smithi09731216-49 2024-04-18T21:53:01.538 INFO:tasks.rados.rados.0.smithi097.stdout:356: oid 49 updating version 0 to 31 2024-04-18T21:53:01.538 INFO:tasks.rados.rados.0.smithi097.stdout:356: oid 49 updating version 31 to 33 2024-04-18T21:53:01.538 INFO:tasks.rados.rados.0.smithi097.stdout:356: oid 49 updating version 33 to 34 2024-04-18T21:53:01.538 INFO:tasks.rados.rados.0.smithi097.stdout:356: oid 49 version 34 is already newer than 32 2024-04-18T21:53:01.538 INFO:tasks.rados.rados.0.smithi097.stdout:356: oid 49 version 34 is already newer than 30 2024-04-18T21:53:01.538 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 49 v 34 (ObjNum 138 snap 27 seq_num 138) dirty exists 2024-04-18T21:53:01.538 INFO:tasks.rados.rados.0.smithi097.stdout:356: left oid 49 (ObjNum 138 snap 27 seq_num 138) 2024-04-18T21:53:01.539 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:01.534+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:01.595 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:01.595 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:01.595 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:53:01.595 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:53:01.595 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:01.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:53:01.657 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:53:01.654+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:53:01.737 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:53:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:53:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:100 3.cs0 3:348e44d8:::smithi09731216-34:head [append 0~557056,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 75.073983085999998, 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50652407300000002, 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100 2024-04-18T21:53:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659953+0000", 2024-04-18T21:53:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.821e-06 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660381+0000", 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042725800000000002 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660395+0000", 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.394e-05 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660404+0000", 2024-04-18T21:53:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6520000000000001e-06 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760564+0000", 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10015995900000001 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760699+0000", 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013526 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762105+0000", 2024-04-18T21:53:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00140595 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965039+0000", 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20293309400000001 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166421+0000", 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201382373 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166476+0000", 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4765999999999997e-05 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:101 3.cs0 3:348e44d8:::smithi09731216-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 75.073513345999999, 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60956795200000002, 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660422+0000", 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-07 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660426+0000", 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3830000000000002e-06 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660436+0000", 2024-04-18T21:53:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8679999999999994e-06 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660443+0000", 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2999999999999998e-06 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762872+0000", 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1024294 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762973+0000", 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001015 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763506+0000", 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053209199999999996 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965047+0000", 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20154100999999999 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269839+0000", 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.304792377 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269989+0000", 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015033399999999999 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:102 3.cs0 3:348e44d8:::smithi09731216-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 75.073463797000002, 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.92759091000000005, 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660472+0000", 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660476+0000", 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0879999999999997e-06 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660493+0000", 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6963e-05 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660502+0000", 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9579999999999996e-06 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763793+0000", 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10329121600000001 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763891+0000", 2024-04-18T21:53:01.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8127000000000004e-05 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372039+0000", 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60814838199999999 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372189+0000", 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014991200000000001 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.588062+0000", 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.215872654 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:78 3.15s0 3:ab33d1b5:::smithi09731216-26:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.752918373, 2024-04-18T21:53:01.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1076682130000002, 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981019+0000", 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.447e-06 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981025+0000", 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6810000000000001e-06 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981044+0000", 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8856999999999999e-05 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981056+0000", 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1921e-05 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164029+0000", 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18297308600000001 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164033+0000", 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0369999999999996e-06 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885490+0000", 2024-04-18T21:53:01.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7214569900000001 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885771+0000", 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028064300000000002 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.088685+0000", 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.202913551 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.759 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:463 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~638976] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.359845524000001, 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50349991299999997, 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 463 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.760 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374091+0000", 2024-04-18T21:53:01.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.84e-06 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375137+0000", 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010459 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375156+0000", 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978e-05 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375169+0000", 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2660000000000001e-05 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571128+0000", 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19595929000000001 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571311+0000", 2024-04-18T21:53:01.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018294599999999999 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572699+0000", 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001388715 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675602+0000", 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102902721 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877476+0000", 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20187386299999999 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877589+0000", 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:464 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~131072] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:01.765 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.358731939000002, 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50427034999999998, 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 464 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375204+0000", 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3650000000000001e-06 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375344+0000", 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013967800000000001 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375357+0000", 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3580999999999999e-05 2024-04-18T21:53:01.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375367+0000", 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0117e-05 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573497+0000", 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198129889 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573606+0000", 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010875 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574480+0000", 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00087374900000000005 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675605+0000", 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10112497400000001 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879333+0000", 2024-04-18T21:53:01.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20372839500000001 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879473+0000", 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013985199999999999 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:465 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.358549404000001, 2024-04-18T21:53:01.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60269858399999998, 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 465 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375386+0000", 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6300000000000004e-07 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375391+0000", 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.729e-06 2024-04-18T21:53:01.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375401+0000", 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9569999999999997e-06 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375409+0000", 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6620000000000005e-06 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777143+0000", 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40173476200000002 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777239+0000", 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5712999999999997e-05 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777695+0000", 2024-04-18T21:53:01.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045595 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.778376+0000", 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068123800000000003 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.977952+0000", 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.199575432 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.978084+0000", 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000132278 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:466 3.cs0 3:3c73a4f8:::smithi09731216-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.358514716999998, 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006726617, 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 466 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375421+0000", 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.009e-06 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375424+0000", 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.495e-06 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375435+0000", 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0360000000000001e-05 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375442+0000", 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6389999999999992e-06 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777929+0000", 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40248657999999998 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777993+0000", 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078687+0000", 2024-04-18T21:53:01.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30069407399999998 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078849+0000", 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162065 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.382147+0000", 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30329795300000001 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.782 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:696 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.256293191999999, 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8198962060000001, 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.783 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 696 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8899999999999998e-07 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477645+0000", 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2030000000000001e-06 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477655+0000", 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8050000000000001e-06 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477663+0000", 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2459999999999997e-06 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.554664+0000", 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.077001179000000003 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:01.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.554666+0000", 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1399999999999998e-06 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.588198+0000", 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0335324410000002 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.588403+0000", 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020478799999999999 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.992774+0000", 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.404371174 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.993448+0000", 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067422999999999995 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.297538+0000", 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30408921100000003 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.788 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:744 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~3768320,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.775514337000001, 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52158899199999997, 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 744 2024-04-18T21:53:01.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958421+0000", 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0499999999999999e-06 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958427+0000", 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0059999999999998e-06 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958440+0000", 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2269999999999999e-05 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958453+0000", 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2893e-05 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.961672+0000", 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032194329999999998 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.962277+0000", 2024-04-18T21:53:01.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060548399999999999 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.480009+0000", 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51773185600000005 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:745 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~3768320,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.775442400999999, 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62853464000000003, 2024-04-18T21:53:01.793 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 745 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:53:01.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958493+0000", 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7700000000000003e-07 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958498+0000", 2024-04-18T21:53:01.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.096e-06 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958513+0000", 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4904e-05 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958522+0000", 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5760000000000006e-06 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.064677+0000", 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106155732 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.065006+0000", 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032830000000000001 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.587027+0000", 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52202115500000001 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.797 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:759 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.690459019, 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62612275900000003, 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.798 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 759 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043479+0000", 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8430000000000001e-06 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043492+0000", 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.322e-05 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043518+0000", 2024-04-18T21:53:01.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6567000000000002e-05 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043533+0000", 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4863e-05 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.201592+0000", 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.158059226 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.201907+0000", 2024-04-18T21:53:01.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031459700000000001 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.669598+0000", 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46769144299999998 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:760 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.690368244, 2024-04-18T21:53:01.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62795852399999996, 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 760 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:53:01.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043567+0000", 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6700000000000002e-07 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043573+0000", 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3499999999999996e-06 2024-04-18T21:53:01.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043585+0000", 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.205e-05 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043594+0000", 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8300000000000002e-06 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.202744+0000", 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15915027200000001 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.202986+0000", 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000241883 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.671525+0000", 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.468539172 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:761 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:01.806 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.69032827, 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.63223503700000006, 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 761 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:53:01.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043607+0000", 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0299999999999998e-07 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043611+0000", 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3030000000000001e-06 2024-04-18T21:53:01.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043621+0000", 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0023999999999999e-05 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043629+0000", 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8460000000000008e-06 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204276+0000", 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16064708899999999 2024-04-18T21:53:01.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204439+0000", 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016309500000000001 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.675841+0000", 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47140197700000003 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.810 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.cs0 94/84 ECSubReadReply(tid=199, attrs_read=0))", 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.529719105, 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46362988399999999, 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304179+0000", 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099963732 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304195+0000", 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5982999999999999e-05 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667760+0000", 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.36356425599999997 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667845+0000", 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5913000000000002e-05 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.cs0 94/84 ECSubRead(tid=200, to_read={3:3c73a4f8:::smithi09731216-45:head=0,356352,0}, subchunks={3:3c73a4f8:::smithi09731216-45:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.529181274000001, 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.463402286, 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304241+0000", 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099488015999999999 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304269+0000", 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7447e-05 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667948+0000", 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.36367946200000001 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.668156+0000", 2024-04-18T21:53:01.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020736100000000001 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.3.19:5 3.5s0 3:a2c4e662:::smithi09731216-50:head [assert-version v34,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:57.306094+0000", 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.4278403319999997, 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41520620600000002, 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.819 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306094+0000", 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306094+0000", 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306097+0000", 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5560000000000002e-06 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306109+0000", 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2327000000000001e-05 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306129+0000", 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0041999999999999e-05 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306143+0000", 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3687e-05 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306283+0000", 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013993 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306491+0000", 2024-04-18T21:53:01.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020857499999999999 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.612374+0000", 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30588252799999999 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.612636+0000", 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026220099999999999 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.721301+0000", 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10866436 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:865 3.5s0 3:a2c4e662:::smithi09731216-50:head [append 0~0] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.425631+0000", 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.308303529, 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41618238899999999, 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.824 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 865 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425631+0000", 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425631+0000", 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425634+0000", 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6630000000000002e-06 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425646+0000", 2024-04-18T21:53:01.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1747e-05 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425678+0000", 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1999000000000003e-05 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425695+0000", 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7635999999999999e-05 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425804+0000", 2024-04-18T21:53:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010906199999999999 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425810+0000", 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4369999999999998e-06 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.526362+0000", 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10055267499999999 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.526579+0000", 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021679199999999999 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.528660+0000", 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002080391 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.630780+0000", 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102120831 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.841660+0000", 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21087966999999999 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.841814+0000", 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015348599999999999 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:866 3.5s0 3:a2c4e662:::smithi09731216-50:head [rollback 1c] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.425732+0000", 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.308202238, 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41635736299999998, 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 866 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.831 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425732+0000", 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425732+0000", 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425733+0000", 2024-04-18T21:53:01.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7700000000000003e-07 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425738+0000", 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3229999999999999e-06 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425749+0000", 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1653e-05 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425758+0000", 2024-04-18T21:53:01.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3860000000000007e-06 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425848+0000", 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0006000000000005e-05 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425851+0000", 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5949999999999999e-06 2024-04-18T21:53:01.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.533729+0000", 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107877722 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.533900+0000", 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017071 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.535750+0000", 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018506180000000001 2024-04-18T21:53:01.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.630783+0000", 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.095032545999999996 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.842013+0000", 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21123041200000001 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.842090+0000", 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6514999999999995e-05 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.836 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:867 3.5s0 3:a2c4e662:::smithi09731216-50:head [append 0~0] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.425773+0000", 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.3081615389999999, 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41649866800000002, 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 867 2024-04-18T21:53:01.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425773+0000", 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425773+0000", 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425774+0000", 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.54e-07 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425777+0000", 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3699999999999999e-06 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425789+0000", 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1833e-05 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425797+0000", 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3059999999999998e-06 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.525877+0000", 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100079977 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.525890+0000", 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2149e-05 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.539816+0000", 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013926122000000001 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.540218+0000", 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040234500000000002 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.541647+0000", 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014284969999999999 2024-04-18T21:53:01.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.630785+0000", 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.089138334 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.842202+0000", 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.211417461 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.842272+0000", 2024-04-18T21:53:01.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9519999999999998e-05 2024-04-18T21:53:01.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.843 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:01.843 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:01.843 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:01.844 INFO:tasks.rados.rados.0.smithi097.stdout:353: finishing write tid 6 to smithi09731216-12 2024-04-18T21:53:01.844 INFO:tasks.rados.rados.0.smithi097.stdout:353: oid 12 updating version 0 to 75 2024-04-18T21:53:01.844 INFO:tasks.rados.rados.0.smithi097.stdout:353: oid 12 updating version 75 to 78 2024-04-18T21:53:01.844 INFO:tasks.rados.rados.0.smithi097.stdout:353: oid 12 version 78 is already newer than 77 2024-04-18T21:53:01.844 INFO:tasks.rados.rados.0.smithi097.stdout:353: oid 12 version 78 is already newer than 74 2024-04-18T21:53:01.845 INFO:tasks.rados.rados.0.smithi097.stdout:353: oid 12 version 78 is already newer than 76 2024-04-18T21:53:01.845 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 12 v 78 (ObjNum 136 snap 27 seq_num 136) dirty exists 2024-04-18T21:53:01.845 INFO:tasks.rados.rados.0.smithi097.stdout:353: left oid 12 (ObjNum 136 snap 27 seq_num 136) 2024-04-18T21:53:01.845 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:53:01.838+0000 7fe581bb5640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 1a after trimming was started 2024-04-18T21:53:01.845 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:53:01.838+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:53:01.846 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:53:01.853 INFO:tasks.rados.rados.0.smithi097.stdout:359: finishing copy_from racing read to smithi09731216-37 2024-04-18T21:53:01.945 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:01.942+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:02.022 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:02.022 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:02.022 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:53:02.022 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:02.036 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:53:02.046 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:02.042+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:02.144 INFO:tasks.rados.rados.0.smithi097.stdout:358: finishing copy_from racing read to smithi09731216-20 2024-04-18T21:53:02.148 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:53:02.142+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:53:02.155 INFO:tasks.rados.rados.0.smithi097.stdout:358: finishing copy_from to smithi09731216-20 2024-04-18T21:53:02.155 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 20 v 181 (ObjNum 126 snap 24 seq_num 126) dirty exists 2024-04-18T21:53:02.155 INFO:tasks.rados.rados.0.smithi097.stdout:359: finishing copy_from to smithi09731216-37 2024-04-18T21:53:02.155 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 37 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-18T21:53:02.208 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:02.208 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:02.208 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:53:02.208 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:53:02.208 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:02.221 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:53:02.248 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:53:02.242+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:53:02.348 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:02.348 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:53:02.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:239 3.as0 3:5bb4af71:::smithi09731216-7:head [delete] snapc 4=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.331048968999994, 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40857630099999998, 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.349 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014032+0000", 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014034+0000", 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2500000000000001e-06 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014044+0000", 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1889999999999996e-06 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.014067+0000", 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3251000000000002e-05 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114145+0000", 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100078193 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114306+0000", 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161044 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.114320+0000", 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4044e-05 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317101+0000", 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20278075300000001 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.317362+0000", 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026128799999999998 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.319467+0000", 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021048719999999998 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.326280+0000", 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006813375 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422400+0000", 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096119880000000005 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:54.422608+0000", 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020816199999999999 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:308 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~737280] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:53:02.355 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 64.254608892999997, 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69365002799999997, 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 308 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090475+0000", 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.976e-06 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091768+0000", 2024-04-18T21:53:02.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00129317 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091803+0000", 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4502999999999998e-05 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091826+0000", 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3235999999999999e-05 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270501+0000", 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.178674888 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270680+0000", 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017917599999999999 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.273556+0000", 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028757610000000001 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281580+0000", 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080240799999999994 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783886+0000", 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50230573000000001 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784122+0000", 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023650799999999999 2024-04-18T21:53:02.360 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:309 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 64.252527900000004, 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69216249600000002, 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.361 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 309 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092555+0000", 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093122+0000", 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056744499999999997 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093146+0000", 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128999999999999e-05 2024-04-18T21:53:02.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093165+0000", 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8300000000000001e-05 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275352+0000", 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18218714899999999 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275632+0000", 2024-04-18T21:53:02.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028059100000000001 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.277549+0000", 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001917087 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281584+0000", 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004034513 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784502+0000", 2024-04-18T21:53:02.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50291762600000001 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784716+0000", 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021415599999999999 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:310 3.as0 3:5bb4af71:::smithi09731216-7:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:53:02.366 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 64.251886729999995, 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69187687499999995, 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 310 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:53:02.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093196+0000", 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093210+0000", 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3517e-05 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093227+0000", 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7050000000000001e-05 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093243+0000", 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5965999999999999e-05 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279008+0000", 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.185765398 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279287+0000", 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027941400000000002 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.280700+0000", 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412581 2024-04-18T21:53:02.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482172+0000", 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20147153900000001 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784956+0000", 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30278430299999998 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785071+0000", 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115454 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:311 3.as0 3:5bb4af71:::smithi09731216-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 64.251811778000004, 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99553992999999996, 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.372 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 311 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093271+0000", 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093278+0000", 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1459999999999999e-06 2024-04-18T21:53:02.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093295+0000", 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7408999999999999e-05 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093328+0000", 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2276000000000001e-05 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785901+0000", 2024-04-18T21:53:02.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.692573628 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.786181+0000", 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027934999999999999 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.088809+0000", 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30262875700000003 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:429 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 47.899405342999998, 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51405278700000001, 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 429 2024-04-18T21:53:02.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445677+0000", 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5260000000000001e-06 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445685+0000", 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445703+0000", 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8295999999999999e-05 2024-04-18T21:53:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445718+0000", 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5383000000000002e-05 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545773+0000", 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100054752 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545785+0000", 2024-04-18T21:53:02.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1783000000000001e-05 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557166+0000", 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011381578 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557443+0000", 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027702800000000002 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.558675+0000", 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012320479999999999 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659884+0000", 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101208162 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959511+0000", 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29962765699999999 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959729+0000", 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021733700000000001 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:495 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~737280] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.838041910000001, 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.664354899, 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.383 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 495 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507043+0000", 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.596e-06 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508389+0000", 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013461460000000001 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508432+0000", 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3294999999999998e-05 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508456+0000", 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4134e-05 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581521+0000", 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.073064403 2024-04-18T21:53:02.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581827+0000", 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030644999999999999 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.586475+0000", 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046472989999999997 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.699953+0000", 2024-04-18T21:53:02.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.113478523 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171243+0000", 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47128973499999999 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171394+0000", 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000151318 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.388 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:496 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~573440] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.834854257000003, 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76183363900000001, 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.389 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 496 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510229+0000", 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.007e-06 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511015+0000", 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000785792 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511046+0000", 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0873000000000001e-05 2024-04-18T21:53:02.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511061+0000", 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5293000000000001e-05 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.691993+0000", 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.180931701 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.692191+0000", 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019870099999999999 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.694402+0000", 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022106690000000002 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.701975+0000", 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075735730000000001 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.271924+0000", 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56994823500000003 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272061+0000", 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013679499999999999 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:497 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~196608] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:53:02.394 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.833804895, 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76114050600000005, 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 497 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511278+0000", 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.387e-06 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511608+0000", 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033003500000000001 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511637+0000", 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9706e-05 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511661+0000", 2024-04-18T21:53:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3439999999999999e-05 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696155+0000", 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18449448800000001 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696457+0000", 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000301496 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.698580+0000", 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021231879999999998 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.702633+0000", 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040534079999999997 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272305+0000", 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56967170199999995 2024-04-18T21:53:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272417+0000", 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111656 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:535 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~802816] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.946596182, 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60500271100000003, 2024-04-18T21:53:02.400 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 535 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398487+0000", 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0650000000000001e-06 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399399+0000", 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091160899999999997 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399417+0000", 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7915999999999999e-05 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399426+0000", 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7070000000000004e-06 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.485946+0000", 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086519677000000003 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.486140+0000", 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019389 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.488798+0000", 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002657707 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.600548+0000", 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11175003 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003305+0000", 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40275720799999998 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003488+0000", 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000182902 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:536 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~638976] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.941618525000003, 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60040761799999998, 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 536 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.09e-07 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404080+0000", 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00061689199999999996 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404092+0000", 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2055000000000001e-05 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404101+0000", 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5650000000000002e-06 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491527+0000", 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087426312000000006 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491697+0000", 2024-04-18T21:53:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017024799999999999 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.493862+0000", 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021641299999999998 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700761+0000", 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20689907199999999 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003718+0000", 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30295725099999998 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003870+0000", 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015228400000000001 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:537 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~131072] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.940648864000003, 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59990886700000001, 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 537 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404433+0000", 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.44e-07 2024-04-18T21:53:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404610+0000", 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017705100000000001 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404620+0000", 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8050000000000001e-06 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404625+0000", 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5029999999999997e-06 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495333+0000", 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.090708141000000006 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495532+0000", 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001983 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.496836+0000", 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013041789999999999 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700772+0000", 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20393593900000001 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004176+0000", 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30340381399999999 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004341+0000", 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016549100000000001 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:538 3.10s0 3:0e754656:::smithi09731216-19:head [setxattr _header (57) in=7b] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.940446876999999, 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70001739100000004, 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 538 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404635+0000", 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6699999999999999e-07 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404637+0000", 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.683e-06 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404642+0000", 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.967e-06 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404647+0000", 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.25e-06 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.497929+0000", 2024-04-18T21:53:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.093281966999999993 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498095+0000", 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016633800000000001 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498905+0000", 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081039199999999995 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700775+0000", 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20186959700000001 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104337+0000", 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40356159400000002 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104652+0000", 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031513599999999999 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:539 3.10s0 3:0e754656:::smithi09731216-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.940428803000003, 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90384237700000003, 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 539 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404653+0000", 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0600000000000001e-07 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404655+0000", 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.95e-06 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404662+0000", 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9530000000000002e-06 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404666+0000", 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4089999999999996e-06 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499383+0000", 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094717379000000004 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499540+0000", 2024-04-18T21:53:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015672099999999999 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.105824+0000", 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60628387699999997 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.106088+0000", 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026422700000000002 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.308495+0000", 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20240645500000001 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:572 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~507904 in=507904b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.811859668, 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60481051399999997, 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 572 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533224+0000", 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.824e-06 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534169+0000", 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094509200000000002 2024-04-18T21:53:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534205+0000", 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5482999999999999e-05 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534228+0000", 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2806000000000002e-05 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634530+0000", 2024-04-18T21:53:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10030222799999999 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634798+0000", 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026831400000000002 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.638371+0000", 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035731249999999999 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744705+0000", 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106333831 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.137882+0000", 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39317701599999999 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.138032+0000", 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014979500000000001 2024-04-18T21:53:02.433 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:573 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~606208 in=606208b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.810818359999999, 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70430669099999998, 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.434 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 573 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:53:02.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534265+0000", 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9800000000000001e-06 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535350+0000", 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010856869999999999 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535392+0000", 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1192999999999997e-05 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535427+0000", 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5559999999999998e-05 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642525+0000", 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107097461 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642617+0000", 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2181000000000004e-05 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.643719+0000", 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011017449999999999 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747184+0000", 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103465025 2024-04-18T21:53:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238444+0000", 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49126059700000002 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238569+0000", 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012526200000000001 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:574 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~557056 in=557056b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.809623420000001, 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70342813999999998, 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 574 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.440 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535460+0000", 2024-04-18T21:53:02.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.796e-06 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540737+0000", 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052778449999999998 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540774+0000", 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6372999999999999e-05 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540799+0000", 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5743999999999999e-05 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744797+0000", 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20399719899999999 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744963+0000", 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016631799999999999 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.746246+0000", 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012830319999999999 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747841+0000", 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001595352 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238775+0000", 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49093394899999998 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238886+0000", 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110532 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:575 3.as0 3:561bcae7:::smithi09731216-2:head [setxattr _header (56) in=63b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.804242967, 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69838045999999998, 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.445 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 575 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:53:02.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540841+0000", 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4540000000000001e-06 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540850+0000", 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2329999999999995e-06 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540874+0000", 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3980000000000001e-05 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540893+0000", 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9173000000000001e-05 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748051+0000", 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20715831100000001 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748150+0000", 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9155999999999997e-05 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748574+0000", 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042389100000000002 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.753028+0000", 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044531889999999998 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.239064+0000", 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.48603637100000002 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.239219+0000", 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154702 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 82/78 [PushReplyOp(3:0e754656:::smithi09731216-19:10)])", 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 23.050238698000001, 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81294819200000001, 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294850+0000", 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6299999999999998e-06 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294862+0000", 2024-04-18T21:53:02.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.186e-05 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294892+0000", 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0103e-05 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294914+0000", 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1633000000000001e-05 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.107740+0000", 2024-04-18T21:53:02.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81282626999999996 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.107791+0000", 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0695999999999997e-05 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 82/78 [PushReplyOp(3:0e754656:::smithi09731216-19:10)])", 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:53:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 23.049359553999999, 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91227883099999996, 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:53:02.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295723+0000", 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.703e-06 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295734+0000", 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.064e-05 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295762+0000", 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7606999999999998e-05 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295776+0000", 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.402e-05 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.207972+0000", 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91219607400000002 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.208000+0000", 2024-04-18T21:53:02.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8787000000000001e-05 2024-04-18T21:53:02.458 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.458 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.458 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.458 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.458 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.458 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:02.460 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:53:02.461 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:02.454+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:353: done (4 left) 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:356: done (3 left) 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:358: done (2 left) 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:359: done (1 left) 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:360: done (0 left) 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:361: read oid 15 snap -1 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:361: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:362: copy_from oid 3 from oid 47 current snap is 28 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 38 current snap is 28 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:363: seq_num 139 ranges {7258112=786432,8044544=688128,8732672=475136,9207808=393216} 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:363: writing smithi09731216-38 from 7258112 to 8044544 tid 1 2024-04-18T21:53:02.462 INFO:tasks.rados.rados.0.smithi097.stdout:363: writing smithi09731216-38 from 8044544 to 8732672 tid 2 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:363: writing smithi09731216-38 from 8732672 to 9207808 tid 3 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:363: writing smithi09731216-38 from 9207808 to 9601024 tid 4 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:364: read oid 39 snap -1 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:364: expect deleted 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:365: rmattr oid 46 current snap is 28 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:365: done (4 left) 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 35 current snap is 28 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:366: seq_num 140 ranges {3751936=802816,4554752=425984,4980736=720896,5701632=409600} 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:366: writing smithi09731216-35 from 3751936 to 4554752 tid 1 2024-04-18T21:53:02.463 INFO:tasks.rados.rados.0.smithi097.stdout:366: writing smithi09731216-35 from 4554752 to 4980736 tid 2 2024-04-18T21:53:02.464 INFO:tasks.rados.rados.0.smithi097.stdout:366: writing smithi09731216-35 from 4980736 to 5701632 tid 3 2024-04-18T21:53:02.464 INFO:tasks.rados.rados.0.smithi097.stdout:366: writing smithi09731216-35 from 5701632 to 6111232 tid 4 2024-04-18T21:53:02.464 INFO:tasks.rados.rados.0.smithi097.stdout:367: read oid 10 snap -1 2024-04-18T21:53:02.464 INFO:tasks.rados.rados.0.smithi097.stdout:367: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-18T21:53:02.465 INFO:tasks.rados.rados.0.smithi097.stdout:368: copy_from oid 41 from oid 16 current snap is 28 2024-04-18T21:53:02.465 INFO:tasks.rados.rados.0.smithi097.stdout:369: read oid 48 snap -1 2024-04-18T21:53:02.465 INFO:tasks.rados.rados.0.smithi097.stdout:369: expect (ObjNum 98 snap 19 seq_num 98) 2024-04-18T21:53:02.465 INFO:tasks.rados.rados.0.smithi097.stdout:370: snap_create 2024-04-18T21:53:02.560 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:02.554+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:02.571 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:53:02.572 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:53:02.572 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-04-18T21:53:02.572 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-18T21:53:02.572 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-18T21:53:02.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok config set filestore_inject_stall 3 2024-04-18T21:53:02.590 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:53:02.586+0000 7ff7545f2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:53:02.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:53:02.635 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:02.635 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:02.635 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-04-18T21:53:02.635 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:02.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:53:02.694 INFO:tasks.rados.rados.0.smithi097.stdout:362: finishing copy_from to smithi09731216-3 2024-04-18T21:53:02.694 INFO:tasks.rados.rados.0.smithi097.stdout:362: got expected ENOENT (src dne) 2024-04-18T21:53:02.695 INFO:tasks.rados.rados.0.smithi097.stdout:362: finishing copy_from racing read to smithi09731216-3 2024-04-18T21:53:02.697 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:53:02.694+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:53:02.705 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:53:02.705 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "" 2024-04-18T21:53:02.706 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:53:02.762 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:53:02.758+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:53:02.777 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:02.777 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:02.777 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:53:02.777 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:53:02.777 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:02.785 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:53:02.785 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:53:02.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:53:02.799 INFO:tasks.rados.rados.0.smithi097.stdout:369: expect (ObjNum 98 snap 19 seq_num 98) 2024-04-18T21:53:02.802 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:53:02.802 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:53:02.818 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:53:02.819 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:53:02.863 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:53:02.858+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:53:02.960 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 90.221507497000005, 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61088117099999995, 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.961 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 35 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735190+0000", 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735191+0000", 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3699999999999999e-07 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735266+0000", 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4856000000000003e-05 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735279+0000", 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.36e-05 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735288+0000", 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0860000000000008e-06 2024-04-18T21:53:02.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242298+0000", 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50700964999999998 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242568+0000", 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027023100000000001 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:02.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.242974+0000", 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040533399999999998 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246576+0000", 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036026919999999998 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345792+0000", 2024-04-18T21:53:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099215503999999996 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.345953+0000", 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161189 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346071+0000", 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118192 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.967 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 90.221395004000001, 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61108955600000003, 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.968 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 36 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735303+0000", 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.23e-07 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735310+0000", 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7560000000000002e-06 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735323+0000", 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735337+0000", 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4049999999999999e-05 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244243+0000", 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50890638099999996 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244422+0000", 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000178495 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.244638+0000", 2024-04-18T21:53:02.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216562 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.247950+0000", 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033121750000000001 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346232+0000", 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098281564000000002 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346324+0000", 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2042999999999999e-05 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.346392+0000", 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8052999999999994e-05 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.974 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 90.221345984999999, 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.71224671699999997, 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 37 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735352+0000", 2024-04-18T21:53:02.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0999999999999998e-07 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735358+0000", 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2639999999999999e-06 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735367+0000", 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-06 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:32.735375+0000", 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5360000000000004e-06 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245345+0000", 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50996983699999998 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.245615+0000", 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027018099999999998 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.246022+0000", 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000407394 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.248536+0000", 2024-04-18T21:53:02.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025132330000000001 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447444+0000", 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198908419 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447532+0000", 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8010000000000006e-05 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:33.447598+0000", 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6383000000000005e-05 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 85.307945472, 2024-04-18T21:53:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61732750999999997, 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 117 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648752+0000", 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648753+0000", 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648758+0000", 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648768+0000", 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.414e-06 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.648777+0000", 2024-04-18T21:53:02.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.363e-06 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863479+0000", 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.214701524 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863596+0000", 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117039 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:02.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.863810+0000", 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021416800000000001 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:37.864846+0000", 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001036319 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265892+0000", 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40104641899999999 2024-04-18T21:53:02.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.265995+0000", 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010209600000000001 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:38.266080+0000", 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5049000000000005e-05 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.650771211000006, 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60955713600000005, 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 161 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305926+0000", 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.305927+0000", 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306056+0000", 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012895300000000001 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.990 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306068+0000", 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306077+0000", 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9430000000000006e-06 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412179+0000", 2024-04-18T21:53:02.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106101709 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412267+0000", 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8251999999999995e-05 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.412388+0000", 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000121136 2024-04-18T21:53:02.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420472+0000", 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080833510000000008 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915141+0000", 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49466944899999998 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915356+0000", 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215305 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915484+0000", 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012713500000000001 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:02.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.650607387999997, 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61031149600000001, 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:02.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 162 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306090+0000", 2024-04-18T21:53:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306091+0000", 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7300000000000005e-07 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306097+0000", 2024-04-18T21:53:02.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6300000000000003e-06 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306106+0000", 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2199999999999998e-06 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306116+0000", 2024-04-18T21:53:02.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8870000000000004e-06 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.413869+0000", 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107753534 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414135+0000", 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026615200000000002 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:02.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.414518+0000", 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038217299999999997 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420492+0000", 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059740879999999998 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.915940+0000", 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49544827600000002 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916143+0000", 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000202752 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916402+0000", 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025901100000000002 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.650571017999994, 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61064147300000005, 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.002 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 163 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306127+0000", 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.06e-07 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306136+0000", 2024-04-18T21:53:03.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6999999999999997e-06 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306146+0000", 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8439999999999993e-06 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306154+0000", 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415593+0000", 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109439392 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.415833+0000", 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000240101 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.416223+0000", 2024-04-18T21:53:03.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000389804 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420508+0000", 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042851699999999996 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916537+0000", 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49602917499999999 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916669+0000", 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179899999999999 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916768+0000", 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8879e-05 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.650532484999999, 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61096294100000004, 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 164 2024-04-18T21:53:03.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306165+0000", 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306166+0000", 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2300000000000001e-07 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306275+0000", 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109749 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306287+0000", 2024-04-18T21:53:03.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1182999999999999e-05 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306298+0000", 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1487e-05 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417044+0000", 2024-04-18T21:53:03.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110745818 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417282+0000", 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000237712 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.417665+0000", 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000383194 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.420523+0000", 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028578010000000001 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.916880+0000", 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49635731900000002 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917054+0000", 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017436200000000001 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.917128+0000", 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3693000000000001e-05 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4165.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.650386490000002, 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70940268200000001, 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4165", 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/55437718", 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 165 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306311+0000", 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306312+0000", 2024-04-18T21:53:03.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9999999999999997e-07 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306317+0000", 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0699999999999997e-06 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306327+0000", 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7030000000000001e-06 2024-04-18T21:53:03.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.306349+0000", 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2160000000000001e-05 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419233+0000", 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112884159 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419499+0000", 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265865 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.419851+0000", 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035253999999999998 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:41.421040+0000", 2024-04-18T21:53:03.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011886380000000001 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015628+0000", 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59458825299999996 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015677+0000", 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8470000000000002e-05 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:42.015714+0000", 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7123999999999999e-05 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:438 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~622592] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.474622986, 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60190392599999998, 2024-04-18T21:53:03.022 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 438 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482075+0000", 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482077+0000", 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3429999999999998e-06 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482901+0000", 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082364099999999996 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482933+0000", 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2128999999999997e-05 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.482949+0000", 2024-04-18T21:53:03.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5872999999999999e-05 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676460+0000", 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19351131799999999 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.676647+0000", 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018723700000000001 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.678595+0000", 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00194812 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.782086+0000", 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10349056299999999 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083745+0000", 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30165924199999999 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.083979+0000", 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023346 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:439 3.fs0 3:fcd1a062:::smithi09731216-16:head [append 0~147456] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:03.028 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.473005501000003, 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70156516499999999, 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 439 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.029 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483692+0000", 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483694+0000", 2024-04-18T21:53:03.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2129999999999998e-06 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483902+0000", 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020762600000000001 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483934+0000", 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2140000000000001e-05 2024-04-18T21:53:03.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483950+0000", 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5792999999999999e-05 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779688+0000", 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29573833799999999 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.779887+0000", 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019900000000000001 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.781116+0000", 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001229071 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.882567+0000", 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101451022 2024-04-18T21:53:03.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185116+0000", 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.302548704 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.185257+0000", 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014125800000000001 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.034 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:440 3.fs0 3:fcd1a062:::smithi09731216-16:head [setxattr _header (57) in=7b] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.472721626000002, 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.10743576, 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 440 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483976+0000", 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483977+0000", 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7000000000000003e-07 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483982+0000", 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4e-06 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.483994+0000", 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.134e-05 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484003+0000", 2024-04-18T21:53:03.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1169999999999993e-06 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186403+0000", 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70240069199999999 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.186624+0000", 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000220523 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.187505+0000", 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088087199999999999 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188877+0000", 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001371723 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591138+0000", 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40226145699999999 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.591412+0000", 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027366599999999999 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:441 3.fs0 3:fcd1a062:::smithi09731216-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.472682382999999, 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.413060682, 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.042 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 441 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484015+0000", 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484016+0000", 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8299999999999996e-07 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484019+0000", 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3069999999999998e-06 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484038+0000", 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978999999999999e-05 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.484047+0000", 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5939999999999994e-06 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188125+0000", 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70407843599999997 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.188355+0000", 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022941899999999999 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692687+0000", 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50433201000000005 2024-04-18T21:53:03.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.692919+0000", 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023258299999999999 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:15.897076+0000", 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20415657100000001 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.047 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:494 3.13s0 3:c8a99f7f:::smithi09731216-31:6 [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 42.460965416000001, 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59079312500000003, 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.048 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 494 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495732+0000", 2024-04-18T21:53:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495734+0000", 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4929999999999999e-06 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495743+0000", 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5629999999999998e-06 2024-04-18T21:53:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495768+0000", 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5147000000000002e-05 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.495784+0000", 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6019e-05 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.596290+0000", 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100505504 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.596489+0000", 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019867499999999999 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886268+0000", 2024-04-18T21:53:03.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.28977927399999998 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.886434+0000", 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000166288 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.086525+0000", 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20009116199999999 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.053 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:563 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~491520] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 32.535300497999998, 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62112892600000003, 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.054 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 563 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421397+0000", 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.421400+0000", 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8729999999999999e-06 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422217+0000", 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081729399999999998 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422245+0000", 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7497000000000001e-05 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422262+0000", 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7601999999999999e-05 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422362+0000", 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9270000000000003e-05 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.422471+0000", 2024-04-18T21:53:03.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109067 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.423731+0000", 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012607779999999999 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.425601+0000", 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018698549999999999 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042390+0000", 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61678826399999997 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042526+0000", 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013642600000000001 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:564 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~589824] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 32.524730484999999, 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.61084178300000003, 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.061 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 564 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431967+0000", 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.431969+0000", 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.64e-06 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432549+0000", 2024-04-18T21:53:03.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00058043400000000001 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432569+0000", 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0203999999999998e-05 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432578+0000", 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4619999999999996e-06 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432649+0000", 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0669999999999999e-05 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.432762+0000", 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113474 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.433682+0000", 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092051399999999999 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.434979+0000", 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012963530000000001 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042699+0000", 2024-04-18T21:53:03.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60771967500000001 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042809+0000", 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110357 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.067 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:565 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~557056] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 32.514639563000003, 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60103031100000004, 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 565 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.068 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442058+0000", 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442060+0000", 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.843e-06 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442728+0000", 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00066799899999999996 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442745+0000", 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6932999999999999e-05 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442753+0000", 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8029999999999997e-06 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.537940+0000", 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.095187356000000001 2024-04-18T21:53:03.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.538060+0000", 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119986 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.539336+0000", 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012755990000000001 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:03.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546080+0000", 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067440950000000003 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.042980+0000", 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49690054099999997 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.043088+0000", 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108156 2024-04-18T21:53:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:566 3.13s0 3:cd8bc96a:::smithi09731216-48:head [append 0~16384] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 32.513930279999997, 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60493431399999997, 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 566 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442767+0000", 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442768+0000", 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5899999999999995e-07 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442790+0000", 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1687000000000001e-05 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442802+0000", 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2564e-05 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442808+0000", 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2830000000000001e-06 2024-04-18T21:53:03.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540321+0000", 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097512961999999995 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540613+0000", 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029227700000000002 2024-04-18T21:53:03.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.542271+0000", 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016580010000000001 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546084+0000", 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038128620000000002 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.047609+0000", 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50152518300000004 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.047702+0000", 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2736000000000001e-05 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:567 3.13s0 3:cd8bc96a:::smithi09731216-48:head [setxattr _header (57) in=7b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 32.513880815999997, 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70082249600000002, 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 567 2024-04-18T21:53:03.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442817+0000", 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9399999999999995e-07 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442820+0000", 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4229999999999999e-06 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442826+0000", 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0440000000000001e-06 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442830+0000", 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0640000000000004e-06 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.543400+0000", 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100569913 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.543930+0000", 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052991099999999999 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.545388+0000", 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014587059999999999 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.646614+0000", 2024-04-18T21:53:03.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101225551 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.143425+0000", 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.496811477 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.143639+0000", 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021391300000000001 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:568 3.13s0 3:cd8bc96a:::smithi09731216-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 32.513862570999997, 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90286167399999995, 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 568 2024-04-18T21:53:03.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442835+0000", 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999998e-07 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442837+0000", 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8020000000000001e-06 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:03.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442844+0000", 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5869999999999998e-06 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.442848+0000", 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9929999999999997e-06 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.545932+0000", 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103084052 2024-04-18T21:53:03.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.546068+0000", 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013653800000000001 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.144727+0000", 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59865855499999998 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.144992+0000", 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026491399999999999 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.345697+0000", 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.200704883 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:03.092 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:03.093 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:03.094 INFO:tasks.rados.rados.0.smithi097.stdout:367: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-18T21:53:03.094 INFO:tasks.rados.rados.0.smithi097.stdout:366: finishing write tid 1 to smithi09731216-35 2024-04-18T21:53:03.094 INFO:tasks.rados.rados.0.smithi097.stdout:366: finishing write tid 2 to smithi09731216-35 2024-04-18T21:53:03.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:53:03.096 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:53:03.090+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:53:03.098 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:53:03.099 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:53:03.130 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:53:03.133 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:53:03.163 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:53:03.163 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:53:03.181 INFO:tasks.rados.rados.0.smithi097.stdout:366: finishing write tid 3 to smithi09731216-35 2024-04-18T21:53:03.181 INFO:tasks.rados.rados.0.smithi097.stdout:366: finishing write tid 4 to smithi09731216-35 2024-04-18T21:53:03.182 INFO:tasks.rados.rados.0.smithi097.stdout:366: finishing write tid 5 to smithi09731216-35 2024-04-18T21:53:03.195 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:03.190+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:03.229 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-18T21:53:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-04-18T21:53:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-04-18T21:53:03.230 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-18T21:53:03.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:53:03.281 INFO:tasks.rados.rados.0.smithi097.stdout:363: finishing write tid 1 to smithi09731216-38 2024-04-18T21:53:03.281 INFO:tasks.rados.rados.0.smithi097.stdout:363: finishing write tid 2 to smithi09731216-38 2024-04-18T21:53:03.282 INFO:tasks.rados.rados.0.smithi097.stdout:363: finishing write tid 3 to smithi09731216-38 2024-04-18T21:53:03.282 INFO:tasks.rados.rados.0.smithi097.stdout:363: finishing write tid 4 to smithi09731216-38 2024-04-18T21:53:03.282 INFO:tasks.rados.rados.0.smithi097.stdout:363: finishing write tid 5 to smithi09731216-38 2024-04-18T21:53:03.284 INFO:tasks.rados.rados.0.smithi097.stdout:368: finishing copy_from racing read to smithi09731216-41 2024-04-18T21:53:03.288 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:53:03.288 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:53:03.296 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:53:03.294+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:53:03.317 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:53:03.317 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:53:03.354 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:53:03.354 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:53:03.370 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:03.370 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:03.371 INFO:tasks.rados.rados.0.smithi097.stdout:362: done (8 left) 2024-04-18T21:53:03.371 INFO:tasks.rados.rados.0.smithi097.stdout:364: done (7 left) 2024-04-18T21:53:03.371 INFO:tasks.rados.rados.0.smithi097.stdout:367: done (6 left) 2024-04-18T21:53:03.371 INFO:tasks.rados.rados.0.smithi097.stdout:369: done (5 left) 2024-04-18T21:53:03.372 INFO:tasks.rados.rados.0.smithi097.stdout:370: done (4 left) 2024-04-18T21:53:03.372 INFO:tasks.rados.rados.0.smithi097.stdout:371: read oid 29 snap -1 2024-04-18T21:53:03.372 INFO:tasks.rados.rados.0.smithi097.stdout:371: expect (ObjNum 126 snap 24 seq_num 126) 2024-04-18T21:53:03.372 INFO:tasks.rados.rados.0.smithi097.stdout:372: snap_create 2024-04-18T21:53:03.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:53:03.392 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:53:03.392 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:53:03.397 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:03.394+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:03.419 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:53:03.420 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:53:03.444 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:53:03.445 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:53:03.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-18T21:53:03.486 INFO:tasks.rados.rados.0.smithi097.stdout:361: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-18T21:53:03.495 INFO:tasks.rados.rados.0.smithi097.stdout:363: finishing write tid 6 to smithi09731216-38 2024-04-18T21:53:03.495 INFO:tasks.rados.rados.0.smithi097.stdout:363: oid 38 updating version 0 to 211 2024-04-18T21:53:03.495 INFO:tasks.rados.rados.0.smithi097.stdout:363: oid 38 updating version 211 to 212 2024-04-18T21:53:03.495 INFO:tasks.rados.rados.0.smithi097.stdout:363: oid 38 updating version 212 to 214 2024-04-18T21:53:03.495 INFO:tasks.rados.rados.0.smithi097.stdout:363: oid 38 version 214 is already newer than 210 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:363: oid 38 version 214 is already newer than 213 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 38 v 214 (ObjNum 139 snap 28 seq_num 139) dirty exists 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:363: left oid 38 (ObjNum 139 snap 28 seq_num 139) 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:366: finishing write tid 6 to smithi09731216-35 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:366: oid 35 updating version 0 to 54 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:366: oid 35 version 54 is already newer than 52 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:366: oid 35 updating version 54 to 55 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:366: oid 35 version 55 is already newer than 53 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:366: oid 35 version 55 is already newer than 51 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 35 v 55 (ObjNum 140 snap 28 seq_num 140) dirty exists 2024-04-18T21:53:03.496 INFO:tasks.rados.rados.0.smithi097.stdout:366: left oid 35 (ObjNum 140 snap 28 seq_num 140) 2024-04-18T21:53:03.499 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:53:03.494+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:53:03.507 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:03.507 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:03.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:53:03.589 INFO:tasks.rados.rados.0.smithi097.stdout:368: finishing copy_from to smithi09731216-41 2024-04-18T21:53:03.589 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 41 v 34 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-04-18T21:53:03.599 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:03.594+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:03.646 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:03.646 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:03.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:53:03.681 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:53:03.678+0000 7fe586bbf640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-04-18T21:53:03.701 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:03.698+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:03.779 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:03.780 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:03.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:53:03.802 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:03.798+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:03.902 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:53:03.898+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:53:03.906 INFO:tasks.rados.rados.0.smithi097.stdout:371: expect (ObjNum 126 snap 24 seq_num 126) 2024-04-18T21:53:03.917 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:03.918 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:03.931 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:53:04.003 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:03.998+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:04.091 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:04.091 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:04.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:53:04.105 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:53:04.098+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:53:04.205 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:04.202+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:04.228 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:04.228 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:04.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:53:04.306 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:04.302+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:04.365 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:04.365 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:04.370 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is set 2024-04-18T21:53:04.376 INFO:tasks.rados.rados.0.smithi097.stdout:361: done (5 left) 2024-04-18T21:53:04.376 INFO:tasks.rados.rados.0.smithi097.stdout:363: done (4 left) 2024-04-18T21:53:04.376 INFO:tasks.rados.rados.0.smithi097.stdout:366: done (3 left) 2024-04-18T21:53:04.376 INFO:tasks.rados.rados.0.smithi097.stdout:368: done (2 left) 2024-04-18T21:53:04.376 INFO:tasks.rados.rados.0.smithi097.stdout:371: done (1 left) 2024-04-18T21:53:04.376 INFO:tasks.rados.rados.0.smithi097.stdout:372: done (0 left) 2024-04-18T21:53:04.376 INFO:tasks.rados.rados.0.smithi097.stdout:373: delete oid 49 current snap is 30 2024-04-18T21:53:04.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:53:04.407 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:04.402+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:04.508 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:53:04.502+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:53:04.550 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:04.550 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:04.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:53:04.609 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:53:04.606+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:53:04.691 INFO:tasks.rados.rados.0.smithi097.stdout:373: done (0 left) 2024-04-18T21:53:04.692 INFO:tasks.rados.rados.0.smithi097.stdout:374: rollback oid 18 current snap is 30 2024-04-18T21:53:04.692 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 18 to 26 2024-04-18T21:53:04.692 INFO:tasks.rados.rados.0.smithi097.stdout:375: snap_create 2024-04-18T21:53:04.710 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:04.706+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:04.726 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:04.726 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:04.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:53:04.793 INFO:tasks.rados.rados.0.smithi097.stdout:374: finishing rollback tid 1 to smithi09731216-18 2024-04-18T21:53:04.793 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 18 v 0 (ObjNum 161 snap 0 seq_num 0) dirty dne 2024-04-18T21:53:04.811 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:04.806+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:04.911 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:04.911 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:04.912 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:53:04.906+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:53:04.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-18T21:53:05.012 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:05.006+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:05.055 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:05.055 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:05.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-18T21:53:05.113 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:53:05.110+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:53:05.214 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:53:05.210+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:53:05.236 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:05.236 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:05.250 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-18T21:53:05.315 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:05.310+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:05.373 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:05.373 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:05.378 INFO:tasks.rados.rados.0.smithi097.stdout:374: done (1 left) 2024-04-18T21:53:05.378 INFO:tasks.rados.rados.0.smithi097.stdout:375: done (0 left) 2024-04-18T21:53:05.378 INFO:tasks.rados.rados.0.smithi097.stdout:376: rollback oid 41 current snap is 31 2024-04-18T21:53:05.378 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 41 to 29 2024-04-18T21:53:05.378 INFO:tasks.rados.rados.0.smithi097.stdout:377: rollback oid 18 current snap is 31 2024-04-18T21:53:05.379 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 18 to 30 2024-04-18T21:53:05.379 INFO:tasks.rados.rados.0.smithi097.stdout:378: rmattr oid 23 current snap is 31 2024-04-18T21:53:05.379 INFO:tasks.rados.rados.0.smithi097.stdout:378: done (2 left) 2024-04-18T21:53:05.379 INFO:tasks.rados.rados.0.smithi097.stdout:379: copy_from oid 43 from oid 10 current snap is 31 2024-04-18T21:53:05.379 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 19 current snap is 31 2024-04-18T21:53:05.379 INFO:tasks.rados.rados.0.smithi097.stdout:380: seq_num 142 ranges {2080768=442368,2523136=720896,3244032=540672,3784704=688128} 2024-04-18T21:53:05.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-18T21:53:05.399 INFO:tasks.rados.rados.0.smithi097.stdout:380: writing smithi09731216-19 from 2080768 to 2523136 tid 1 2024-04-18T21:53:05.405 INFO:tasks.rados.rados.0.smithi097.stdout:380: writing smithi09731216-19 from 2523136 to 3244032 tid 2 2024-04-18T21:53:05.410 INFO:tasks.rados.rados.0.smithi097.stdout:380: writing smithi09731216-19 from 3244032 to 3784704 tid 3 2024-04-18T21:53:05.416 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:53:05.410+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:53:05.416 INFO:tasks.rados.rados.0.smithi097.stdout:380: writing smithi09731216-19 from 3784704 to 4472832 tid 4 2024-04-18T21:53:05.416 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 13 current snap is 31 2024-04-18T21:53:05.416 INFO:tasks.rados.rados.0.smithi097.stdout:381: seq_num 143 ranges {2015232=458752,2473984=475136,2949120=655360,3604480=671744,4276224=147456} 2024-04-18T21:53:05.436 INFO:tasks.rados.rados.0.smithi097.stdout:381: writing smithi09731216-13 from 2015232 to 2473984 tid 1 2024-04-18T21:53:05.439 INFO:tasks.rados.rados.0.smithi097.stdout:381: writing smithi09731216-13 from 2473984 to 2949120 tid 2 2024-04-18T21:53:05.444 INFO:tasks.rados.rados.0.smithi097.stdout:381: writing smithi09731216-13 from 2949120 to 3604480 tid 3 2024-04-18T21:53:05.449 INFO:tasks.rados.rados.0.smithi097.stdout:381: writing smithi09731216-13 from 3604480 to 4276224 tid 4 2024-04-18T21:53:05.450 INFO:tasks.rados.rados.0.smithi097.stdout:381: writing smithi09731216-13 from 4276224 to 4423680 tid 5 2024-04-18T21:53:05.450 INFO:tasks.rados.rados.0.smithi097.stdout:382: setattr oid 26 current snap is 31 2024-04-18T21:53:05.452 INFO:tasks.rados.rados.0.smithi097.stdout:383: snap_create 2024-04-18T21:53:05.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-18T21:53:05.516 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:05.516 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:05.516 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:05.510+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:05.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-18T21:53:05.585 INFO:tasks.rados.rados.0.smithi097.stdout:379: finishing copy_from racing read to smithi09731216-43 2024-04-18T21:53:05.616 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:53:05.616 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:53:05.617 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:05.614+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:05.652 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:05.652 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:05.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-18T21:53:05.705 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:53:05.705 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:53:05.718 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:53:05.714+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:53:05.730 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:53:05.730 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:53:05.746 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:53:05.746 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:53:05.760 INFO:tasks.rados.rados.0.smithi097.stdout:380: finishing write tid 1 to smithi09731216-19 2024-04-18T21:53:05.768 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:53:05.768 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:53:05.786 INFO:tasks.rados.rados.0.smithi097.stdout:379: finishing copy_from to smithi09731216-43 2024-04-18T21:53:05.786 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 43 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-18T21:53:05.789 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:53:05.790 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:53:05.819 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:53:05.814+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:53:05.822 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:05.822 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:94 3.0s0 3:033c99c3:::smithi09731216-32:head [append 0~524288,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 79.265691587000006, 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40893576500000001, 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 94 2024-04-18T21:53:05.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555850+0000", 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.555853+0000", 2024-04-18T21:53:05.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1889999999999999e-06 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556617+0000", 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076362200000000004 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556649+0000", 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2478999999999999e-05 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556665+0000", 2024-04-18T21:53:05.825 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5837000000000001e-05 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556826+0000", 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016137200000000001 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.557241+0000", 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00041440699999999999 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.559642+0000", 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024011430000000001 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563415+0000", 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003773233 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964577+0000", 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40116224099999997 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.964786+0000", 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000208242 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:95 3.0s0 3:033c99c3:::smithi09731216-32:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 79.264838490000002, 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40858905099999998, 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.828 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 95 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556703+0000", 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556705+0000", 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6360000000000001e-06 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556715+0000", 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.066e-05 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556734+0000", 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8323999999999999e-05 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556745+0000", 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1240000000000001e-05 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561327+0000", 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0045823330000000001 2024-04-18T21:53:05.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.561583+0000", 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000255732 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.562607+0000", 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010240990000000001 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.563867+0000", 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012605439999999999 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965078+0000", 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40121052699999998 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.965292+0000", 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021395600000000001 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.833 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:96 3.0s0 3:033c99c3:::smithi09731216-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 79.264740947999996, 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42323290800000002, 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.834 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 96 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556800+0000", 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556802+0000", 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7400000000000001e-06 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556808+0000", 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.993e-06 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.556828+0000", 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9364e-05 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.656965+0000", 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10013702300000001 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762039+0000", 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105074903 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.762350+0000", 2024-04-18T21:53:05.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031089399999999998 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.967780+0000", 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20542997700000001 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.972542+0000", 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004761932 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.838 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.979990+0000", 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0074475289999999996 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.980033+0000", 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3553000000000001e-05 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.839 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:224 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 72.808408838000005, 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40564824999999999, 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 224 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013133+0000", 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013134+0000", 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.387e-06 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013144+0000", 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0273e-05 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013166+0000", 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1857000000000001e-05 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.013182+0000", 2024-04-18T21:53:05.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5645999999999999e-05 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113542+0000", 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10035996599999999 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.113552+0000", 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0492999999999999e-05 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.843 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215576+0000", 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102023869 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.215730+0000", 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000153635 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:53.418781+0000", 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203051124 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.844 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:302 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 67.759890517000002, 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.406305586, 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.845 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 302 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061651+0000", 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061654+0000", 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2830000000000002e-06 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061669+0000", 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4637000000000001e-05 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061699+0000", 2024-04-18T21:53:05.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9791e-05 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061714+0000", 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5441999999999999e-05 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061811+0000", 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7153999999999998e-05 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061819+0000", 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9279999999999993e-06 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064181+0000", 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002361447 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.064306+0000", 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012514399999999999 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.065234+0000", 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092848599999999996 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.165102+0000", 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.099867808000000002 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467877+0000", 2024-04-18T21:53:05.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.302775085 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.467956+0000", 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9381000000000005e-05 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:303 3.0s0 3:00e42e84:::smithi09731216-41:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 67.759791127, 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50679023400000001, 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 303 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.852 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061750+0000", 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061751+0000", 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5499999999999998e-07 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061754+0000", 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2339999999999999e-06 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061763+0000", 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3950000000000007e-06 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061769+0000", 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9730000000000002e-06 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061904+0000", 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013470999999999999 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061911+0000", 2024-04-18T21:53:05.855 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2200000000000003e-06 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163859+0000", 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101947416 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.163977+0000", 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117971 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.164692+0000", 2024-04-18T21:53:05.856 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00071513499999999999 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267070+0000", 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102378067 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568454+0000", 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.301384333 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.857 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.568541+0000", 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6224999999999995e-05 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:304 3.0s0 3:00e42e84:::smithi09731216-41:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 67.759748654999996, 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60693648, 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.858 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 304 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.859 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061793+0000", 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3000000000000001e-07 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061797+0000", 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3680000000000002e-06 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061804+0000", 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0809999999999996e-06 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061809+0000", 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5280000000000003e-06 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061950+0000", 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000140529 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.061958+0000", 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5329999999999994e-06 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166016+0000", 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.104057925 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166118+0000", 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010206500000000001 2024-04-18T21:53:05.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.166690+0000", 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057174600000000004 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.267073+0000", 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100382727 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668597+0000", 2024-04-18T21:53:05.863 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401524149 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.668729+0000", 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132399 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:332 3.0s0 3:00b6e030:::smithi09731216-21:head [copy-from ver 39] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.615589675999999, 2024-04-18T21:53:05.864 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.72786893100000005, 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 332 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:53:05.865 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205952+0000", 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205954+0000", 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4370000000000001e-06 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.866 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.205969+0000", 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4516e-05 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206001+0000", 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.252e-05 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206022+0000", 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.084e-05 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306220+0000", 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100197961 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306483+0000", 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026249200000000002 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.726594+0000", 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42011143499999998 2024-04-18T21:53:05.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.730261+0000", 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036666060000000002 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933714+0000", 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.203453467 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.869 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.933821+0000", 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010665700000000001 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:333 3.0s0 3:00b6e030:::smithi09731216-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.615484703000007, 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52393701999999998, 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:05.870 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 333 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206057+0000", 2024-04-18T21:53:05.871 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206058+0000", 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.426e-06 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206065+0000", 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4799999999999998e-06 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.872 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206085+0000", 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0089999999999999e-05 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.206101+0000", 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.643e-05 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306839+0000", 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100738203 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.873 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.306987+0000", 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014755599999999999 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729615+0000", 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42262781100000002 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729821+0000", 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020588799999999999 2024-04-18T21:53:05.874 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.729994+0000", 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000173136 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:430 3.0s0 3:00b6e030:::smithi09731216-21:head [copy-from ver 39] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 51.375815183, 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.71883465800000002, 2024-04-18T21:53:05.875 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 430 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445726+0000", 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445730+0000", 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7629999999999998e-06 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445749+0000", 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8960000000000001e-05 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445786+0000", 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7049999999999999e-05 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445804+0000", 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8236e-05 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445984+0000", 2024-04-18T21:53:05.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017996200000000001 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446012+0000", 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7554000000000001e-05 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.515978+0000", 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069965816 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516184+0000", 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020612199999999999 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.970820+0000", 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45463621900000001 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.978038+0000", 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0072179430000000001 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.164404+0000", 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18636640400000001 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:15.164561+0000", 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015662900000000001 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:431 3.0s0 3:00b6e030:::smithi09731216-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 51.375689430999998, 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.52894810699999995, 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 431 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.882 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445852+0000", 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445853+0000", 2024-04-18T21:53:05.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.437e-06 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445863+0000", 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4769999999999992e-06 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445885+0000", 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2093e-05 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.445901+0000", 2024-04-18T21:53:05.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5826999999999999e-05 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446183+0000", 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028208399999999999 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.446201+0000", 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8133000000000001e-05 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516497+0000", 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.070296147000000003 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for blocked object", 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.516639+0000", 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000141533 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974265+0000", 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.45762602699999999 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.886 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974495+0000", 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022996099999999999 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:14.974800+0000", 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030538800000000002 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:560 3.0s0 3:00e42e84:::smithi09731216-41:head [copy-from ver 4] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:05.887 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 35.803114958999998, 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41315372, 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 560 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:53:05.888 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018426+0000", 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018429+0000", 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.357e-06 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.889 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018443+0000", 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4491999999999999e-05 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018468+0000", 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5060000000000001e-05 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.018488+0000", 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9259e-05 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.890 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.118707+0000", 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10021904099999999 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.118865+0000", 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015874199999999999 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.226410+0000", 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107544398 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.891 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.229100+0000", 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026898130000000001 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.431493+0000", 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20239311800000001 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.431580+0000", 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7440000000000003e-05 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:646 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~180224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:44.420715+0000", 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.400826243000001, 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8292185139999999, 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 646 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.893 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420715+0000", 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420715+0000", 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420717+0000", 2024-04-18T21:53:05.894 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7719999999999999e-06 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420727+0000", 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5640000000000002e-06 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420746+0000", 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9561000000000001e-05 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420763+0000", 2024-04-18T21:53:05.895 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6963e-05 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420878+0000", 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114722 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420894+0000", 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6271000000000001e-05 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.523662+0000", 2024-04-18T21:53:05.896 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10276816499999999 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.523665+0000", 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.303e-06 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.941458+0000", 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.417792162 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.941926+0000", 2024-04-18T21:53:05.897 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046790699999999999 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:46.249934+0000", 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30800812399999999 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:647 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:44.420795+0000", 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.400746675000001, 2024-04-18T21:53:05.898 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.930867406, 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 647 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420795+0000", 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.899 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420795+0000", 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420795+0000", 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4600000000000004e-07 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420800+0000", 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4610000000000001e-06 2024-04-18T21:53:05.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420812+0000", 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2296000000000001e-05 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420819+0000", 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7769999999999997e-06 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420967+0000", 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014761900000000001 2024-04-18T21:53:05.901 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420976+0000", 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6500000000000008e-06 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.623739+0000", 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.2027632 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.623746+0000", 2024-04-18T21:53:05.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4840000000000001e-06 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:46.047556+0000", 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4238097139999999 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:46.047653+0000", 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7645000000000004e-05 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:46.351662+0000", 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30400891400000002 2024-04-18T21:53:05.903 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:648 3.0s0 3:00e42e84:::smithi09731216-41:head [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:44.420858+0000", 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.400683445999999, 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9317984240000001, 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.904 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 648 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420858+0000", 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420858+0000", 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420859+0000", 2024-04-18T21:53:05.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2079999999999999e-06 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420865+0000", 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1700000000000002e-06 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420879+0000", 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3752e-05 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.420889+0000", 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7869999999999996e-06 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.421034+0000", 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014515400000000001 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.421049+0000", 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5404999999999999e-05 2024-04-18T21:53:05.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.623794+0000", 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.202744234 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.623795+0000", 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2100000000000001e-06 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:46.047986+0000", 2024-04-18T21:53:05.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4241907620000001 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:46.048129+0000", 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014317400000000001 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:46.352656+0000", 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30452756800000003 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:868 3.0s0 3:00b6e030:::smithi09731216-21:head [append 0~720896] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.432592+0000", 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.3889494850000004, 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41661230999999999, 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.910 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 868 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.432592+0000", 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.432592+0000", 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.911 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.432595+0000", 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6189999999999998e-06 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.433065+0000", 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00047053200000000001 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.433082+0000", 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6731e-05 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.433093+0000", 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.114e-05 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.433425+0000", 2024-04-18T21:53:05.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033218400000000002 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.433552+0000", 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012731600000000001 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.436453+0000", 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002900789 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.538021+0000", 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101567676 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.849003+0000", 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31098161499999999 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.849204+0000", 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020170799999999999 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.915 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:869 3.0s0 3:00b6e030:::smithi09731216-21:head [append 0~622592] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.439012+0000", 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.3825289830000003, 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.411940629, 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.916 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 869 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.439012+0000", 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.439012+0000", 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.439015+0000", 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1040000000000001e-06 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.439345+0000", 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000330868 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.439365+0000", 2024-04-18T21:53:05.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9360000000000001e-05 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.539442+0000", 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007701500000001 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.539504+0000", 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2527000000000006e-05 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.539632+0000", 2024-04-18T21:53:05.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000127661 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540899+0000", 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001267333 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.646653+0000", 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105753862 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.850738+0000", 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20408527200000001 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.850953+0000", 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000214627 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:871 3.0s0 3:00b6e030:::smithi09731216-21:head [append 0~245760] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.539901+0000", 2024-04-18T21:53:05.921 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.2816403349999996, 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41017221500000001, 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 871 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.539901+0000", 2024-04-18T21:53:05.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.539901+0000", 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.539902+0000", 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9400000000000005e-07 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540046+0000", 2024-04-18T21:53:05.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014471199999999999 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540055+0000", 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6330000000000002e-06 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540060+0000", 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0259999999999998e-06 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.644024+0000", 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.103964342 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.644225+0000", 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020028400000000001 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.645546+0000", 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0013209300000000001 2024-04-18T21:53:05.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.847936+0000", 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20239052900000001 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.949915+0000", 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101979296 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.950073+0000", 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015786899999999999 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:872 3.0s0 3:00b6e030:::smithi09731216-21:head [setxattr _header (57) in=7b] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.540067+0000", 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.2814743980000003, 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51091202099999999, 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 872 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540067+0000", 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540067+0000", 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540067+0000", 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7099999999999997e-07 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540070+0000", 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2280000000000001e-06 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540075+0000", 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.011e-06 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540077+0000", 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8270000000000002e-06 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.746858+0000", 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20678094599999999 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.746937+0000", 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8634000000000006e-05 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.747236+0000", 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029944699999999999 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.848174+0000", 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10093725000000001 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.050783+0000", 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20260892599999999 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.050979+0000", 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000196381 2024-04-18T21:53:05.932 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:873 3.0s0 3:00b6e030:::smithi09731216-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.540082+0000", 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.2814598500000001, 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.81387967400000005, 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:05.933 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 873 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540082+0000", 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540082+0000", 2024-04-18T21:53:05.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540082+0000", 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7300000000000002e-07 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540083+0000", 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2860000000000001e-06 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:05.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.540087+0000", 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8589999999999998e-06 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.640187+0000", 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100100151 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.949305+0000", 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30911785899999999 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:05.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.949470+0000", 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000165432 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.052046+0000", 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10257601199999999 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.052328+0000", 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028142400000000001 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:05.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:05.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.353961+0000", 2024-04-18T21:53:05.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30163337800000001 2024-04-18T21:53:05.938 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.938 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.938 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.938 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:05.938 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:05.938 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:05.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-18T21:53:05.940 INFO:tasks.rados.rados.0.smithi097.stdout:376: finishing rollback tid 0 to smithi09731216-41 2024-04-18T21:53:05.940 INFO:tasks.rados.rados.0.smithi097.stdout:376: finishing rollback tid 1 to smithi09731216-41 2024-04-18T21:53:05.940 INFO:tasks.rados.rados.0.smithi097.stdout:376: finishing rollback tid 2 to smithi09731216-41 2024-04-18T21:53:05.941 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 41 v 68 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-04-18T21:53:05.941 INFO:tasks.rados.rados.0.smithi097.stdout:381: finishing write tid 1 to smithi09731216-13 2024-04-18T21:53:05.941 INFO:tasks.rados.rados.0.smithi097.stdout:380: finishing write tid 2 to smithi09731216-19 2024-04-18T21:53:05.941 INFO:tasks.rados.rados.0.smithi097.stdout:377: finishing rollback tid 1 to smithi09731216-18 2024-04-18T21:53:05.941 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 18 v 0 (ObjNum 65 snap 0 seq_num 1635019372) dirty dne 2024-04-18T21:53:05.941 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:05.934+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:05.955 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 26 v 17 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-18T21:53:05.959 INFO:tasks.rados.rados.0.smithi097.stdout:381: finishing write tid 2 to smithi09731216-13 2024-04-18T21:53:05.959 INFO:tasks.rados.rados.0.smithi097.stdout:381: finishing write tid 3 to smithi09731216-13 2024-04-18T21:53:05.959 INFO:tasks.rados.rados.0.smithi097.stdout:381: finishing write tid 4 to smithi09731216-13 2024-04-18T21:53:05.992 INFO:tasks.rados.rados.0.smithi097.stdout:380: finishing write tid 3 to smithi09731216-19 2024-04-18T21:53:06.041 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:06.038+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:06.082 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:06.082 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:53:06.082 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:53:06.082 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:06.093 INFO:tasks.rados.rados.0.smithi097.stdout:380: finishing write tid 4 to smithi09731216-19 2024-04-18T21:53:06.093 INFO:tasks.rados.rados.0.smithi097.stdout:380: finishing write tid 5 to smithi09731216-19 2024-04-18T21:53:06.093 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:53:06.094 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:53:06.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-18T21:53:06.120 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:53:06.121 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:53:06.141 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:06.138+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:06.147 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:53:06.147 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:53:06.174 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:53:06.174 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:53:06.204 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:53:06.204 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:53:06.233 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:53:06.233 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:53:06.237 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:06.237 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:53:06.237 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:53:06.238 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:53:06.238 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:06.242 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:53:06.238+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:53:06.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-18T21:53:06.262 INFO:tasks.rados.rados.0.smithi097.stdout:381: finishing write tid 5 to smithi09731216-13 2024-04-18T21:53:06.262 INFO:tasks.rados.rados.0.smithi097.stdout:381: finishing write tid 6 to smithi09731216-13 2024-04-18T21:53:06.296 INFO:tasks.rados.rados.0.smithi097.stdout:380: finishing write tid 6 to smithi09731216-19 2024-04-18T21:53:06.297 INFO:tasks.rados.rados.0.smithi097.stdout:380: oid 19 updating version 0 to 73 2024-04-18T21:53:06.297 INFO:tasks.rados.rados.0.smithi097.stdout:380: oid 19 version 73 is already newer than 71 2024-04-18T21:53:06.297 INFO:tasks.rados.rados.0.smithi097.stdout:380: oid 19 updating version 73 to 75 2024-04-18T21:53:06.297 INFO:tasks.rados.rados.0.smithi097.stdout:380: oid 19 version 75 is already newer than 74 2024-04-18T21:53:06.297 INFO:tasks.rados.rados.0.smithi097.stdout:380: oid 19 version 75 is already newer than 72 2024-04-18T21:53:06.297 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 19 v 75 (ObjNum 142 snap 31 seq_num 142) dirty exists 2024-04-18T21:53:06.297 INFO:tasks.rados.rados.0.smithi097.stdout:380: left oid 19 (ObjNum 142 snap 31 seq_num 142) 2024-04-18T21:53:06.342 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:06.338+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:06.386 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:06.386 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:53:06.386 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:84 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 80.333583907999994, 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.59192865299999997, 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.387 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 84 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050853+0000", 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050854+0000", 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1399999999999998e-07 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050855+0000", 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5719999999999999e-06 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050860+0000", 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.904e-06 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.050864+0000", 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5750000000000001e-06 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138083+0000", 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087218981000000001 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.138226+0000", 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000143545 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339757+0000", 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20153072 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.339856+0000", 2024-04-18T21:53:06.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9247999999999996e-05 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.642782+0000", 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30292579400000003 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:315 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 68.291660585000002, 2024-04-18T21:53:06.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70415003200000004, 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 315 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092777+0000", 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092778+0000", 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.725e-06 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092785+0000", 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7950000000000002e-06 2024-04-18T21:53:06.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092802+0000", 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6852999999999999e-05 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092810+0000", 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7950000000000008e-06 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152681+0000", 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059871316000000001 2024-04-18T21:53:06.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.152755+0000", 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4179999999999998e-05 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153317+0000", 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00056207900000000005 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253434+0000", 2024-04-18T21:53:06.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100116332 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796715+0000", 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54328157899999996 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.796927+0000", 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021137799999999999 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:316 3.es0 3:7b3c3586:::smithi09731216-12:head [rollback 2] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 68.291611841000005, 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70474535999999999, 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 316 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092825+0000", 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092826+0000", 2024-04-18T21:53:06.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5999999999999999e-07 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092827+0000", 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714e-06 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092833+0000", 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0989999999999998e-06 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092836+0000", 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.056e-06 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.153954+0000", 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061118211999999998 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154059+0000", 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010548399999999999 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.154712+0000", 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065292300000000002 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253436+0000", 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.098723985 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797355+0000", 2024-04-18T21:53:06.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.543919245 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797571+0000", 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021528199999999999 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:317 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~0] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 68.291571438000005, 2024-04-18T21:53:06.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70519442399999999, 2024-04-18T21:53:06.404 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.404 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.404 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.405 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.405 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.405 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.405 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 317 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092866+0000", 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8700000000000002e-07 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092868+0000", 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1440000000000001e-06 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.092873+0000", 2024-04-18T21:53:06.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0860000000000001e-06 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192949+0000", 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007585099999999 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252838+0000", 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059889100000000001 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.252927+0000", 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8651000000000002e-05 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.253260+0000", 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033289600000000001 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.353758+0000", 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100498198 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.797947+0000", 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.444189102 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.798060+0000", 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113109 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.410 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:477 3.es0 3:77cf8780:::smithi09731216-28:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.365379054000002, 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40712708600000003, 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 477 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019058+0000", 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019060+0000", 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7859999999999999e-06 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019070+0000", 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0345e-05 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019098+0000", 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7392000000000001e-05 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019115+0000", 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7765e-05 2024-04-18T21:53:06.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.026938+0000", 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0078220470000000004 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.027160+0000", 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000222625 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.027978+0000", 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081757100000000001 2024-04-18T21:53:06.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.106314+0000", 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.078336021000000006 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.426038+0000", 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.31972404500000001 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.426185+0000", 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000147489 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:478 3.es0 3:77cf8780:::smithi09731216-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.365294126000002, 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.709574606, 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:06.416 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 478 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019143+0000", 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019144+0000", 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.206e-06 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019151+0000", 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4819999999999999e-06 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019170+0000", 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8906000000000002e-05 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.019185+0000", 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5116999999999999e-05 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.028449+0000", 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0092640520000000001 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.028689+0000", 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023998100000000001 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.427127+0000", 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39843774999999998 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.427384+0000", 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025700499999999998 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:19.728718+0000", 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30133410700000002 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:579 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 35.641252500999997, 2024-04-18T21:53:06.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40650074000000003, 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 579 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:53:06.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743185+0000", 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743186+0000", 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.16e-07 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743190+0000", 2024-04-18T21:53:06.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5480000000000003e-06 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743199+0000", 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4570000000000005e-06 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743205+0000", 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7959999999999998e-06 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.844198+0000", 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.100992966 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.844801+0000", 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060208199999999999 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:31.149685+0000", 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.304884975 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:580 3.es0 3:7b11f20d:::smithi09731216-4:head [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 35.641221571000003, 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40696587899999997, 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 580 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743216+0000", 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4799999999999998e-07 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743220+0000", 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5829999999999998e-06 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743227+0000", 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.379e-06 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.743233+0000", 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9220000000000002e-06 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.937914+0000", 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19468091200000001 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:30.938494+0000", 2024-04-18T21:53:06.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000580282 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:31.150182+0000", 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21168725299999999 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:650 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~507904] snapc 18=[18,17,16,15,10,a] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-18T21:53:06.430 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:44.450110+0000", 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.934327628999998, 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49543192400000002, 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 650 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.450110+0000", 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.450110+0000", 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.450111+0000", 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0070000000000001e-06 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.450454+0000", 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034327000000000001 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.450466+0000", 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2488e-05 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.450473+0000", 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5180000000000001e-06 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.737411+0000", 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.28693832899999999 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.737608+0000", 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019714799999999999 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.739309+0000", 2024-04-18T21:53:06.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001701106 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.743209+0000", 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038997760000000002 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.945330+0000", 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20212027799999999 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:44.945542+0000", 2024-04-18T21:53:06.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021200400000000001 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 88/84 ECSubWriteReply(tid=99, last_complete=88'62, committed=1, applied=1))", 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:45.050881+0000", 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.333555774000001, 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.600533172, 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.050881+0000", 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.050881+0000", 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.050883+0000", 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2470000000000001e-06 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.050890+0000", 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8700000000000003e-06 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.050911+0000", 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.092e-05 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.050922+0000", 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.13e-05 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.651384+0000", 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60046252499999997 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.651415+0000", 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0309999999999999e-05 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.439 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 88/84 ECSubWriteReply(tid=99, last_complete=88'62, committed=1, applied=1))", 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:45.150608+0000", 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.233828894999998, 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50090609600000002, 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.150608+0000", 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.150608+0000", 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.150610+0000", 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.48e-06 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.150619+0000", 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0469999999999999e-06 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.150646+0000", 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7253000000000001e-05 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.150660+0000", 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.401e-05 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.651484+0000", 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50082380599999998 2024-04-18T21:53:06.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.651514+0000", 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0499999999999999e-05 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 88/84 ECSubWriteReply(tid=98, last_complete=88'61, committed=1, applied=1))", 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:45.152094+0000", 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.232343742000001, 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49902640599999998, 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.152094+0000", 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.152094+0000", 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.152095+0000", 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.64e-06 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.152105+0000", 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7159999999999998e-06 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.152134+0000", 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8966999999999999e-05 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252215+0000", 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10008135899999999 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.651083+0000", 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39886776699999998 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.651120+0000", 2024-04-18T21:53:06.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6956999999999999e-05 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 88/84 ECSubWriteReply(tid=99, last_complete=88'62, committed=1, applied=1))", 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:45.252260+0000", 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.132177670000001, 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49948108600000002, 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.447 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252260+0000", 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252260+0000", 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252261+0000", 2024-04-18T21:53:06.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.046e-06 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252269+0000", 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1140000000000006e-06 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252289+0000", 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0633e-05 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252300+0000", 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0557000000000001e-05 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.751701+0000", 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49940107 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.751741+0000", 2024-04-18T21:53:06.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9666000000000003e-05 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWriteReply(3.es0 88/84 ECSubWriteReply(tid=99, last_complete=88'62, committed=1, applied=1))", 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:45.252412+0000", 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.132024771000001, 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49951857900000002, 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.451 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252412+0000", 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252412+0000", 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252413+0000", 2024-04-18T21:53:06.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8299999999999998e-07 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252416+0000", 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1329999999999999e-06 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252428+0000", 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.182e-05 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.252436+0000", 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6600000000000001e-06 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.751877+0000", 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49944093299999998 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:45.751931+0000", 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3650000000000003e-05 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.454 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.3s6 99/93 ECSubWrite(tid=350, reqid=osd.4.0:349, at_version=0'0, trim_to=0'0, roll_forward_to=98'187))", 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:55.781893+0000", 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.602544267000001, 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41138486000000002, 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.781893+0000", 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.781893+0000", 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.781895+0000", 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6199999999999999e-06 2024-04-18T21:53:06.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.781915+0000", 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0530000000000002e-05 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.781948+0000", 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2616000000000001e-05 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.781964+0000", 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6558999999999998e-05 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.989236+0000", 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.207271809 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.989251+0000", 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.468e-05 2024-04-18T21:53:06.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:56.193175+0000", 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20392469699999999 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:56.193278+0000", 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010234900000000001 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDECSubOpWrite(3.3s6 99/93 ECSubWrite(tid=351, reqid=osd.4.0:351, at_version=99'189, trim_to=0'0, roll_forward_to=98'187))", 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:55.785043+0000", 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 10.599394610000001, 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.408326509, 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.785043+0000", 2024-04-18T21:53:06.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.785043+0000", 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.785043+0000", 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5600000000000001e-07 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.785083+0000", 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9886999999999999e-05 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.785104+0000", 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1092000000000001e-05 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.785119+0000", 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4657e-05 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:56.090146+0000", 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30502722199999999 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:56.090161+0000", 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5515999999999998e-05 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:56.193297+0000", 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10313520599999999 2024-04-18T21:53:06.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:56.193369+0000", 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2472999999999995e-05 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:877 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~770048] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.489502+0000", 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.8949351139999999, 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40761816000000001, 2024-04-18T21:53:06.464 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 877 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.489502+0000", 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.489502+0000", 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.489504+0000", 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.916e-06 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.489957+0000", 2024-04-18T21:53:06.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00045303599999999998 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.489979+0000", 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1532e-05 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.489991+0000", 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2748000000000001e-05 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.490066+0000", 2024-04-18T21:53:06.467 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4312000000000003e-05 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.490193+0000", 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012715 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.491575+0000", 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0013824880000000001 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.725600+0000", 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23402435299999999 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.896988+0000", 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.171388559 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.897120+0000", 2024-04-18T21:53:06.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013206599999999999 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:879 3.es0 3:7b3c3586:::smithi09731216-12:head [append 0~245760] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.595400+0000", 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7890372829999999, 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.85182017099999996, 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.470 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 879 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595400+0000", 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595400+0000", 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595402+0000", 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8780000000000001e-06 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595565+0000", 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016329699999999999 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595581+0000", 2024-04-18T21:53:06.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6203000000000001e-05 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595591+0000", 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5240000000000006e-06 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.899540+0000", 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30394911400000002 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.899717+0000", 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017654500000000001 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.901254+0000", 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015376700000000001 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.904953+0000", 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003698598 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.447162+0000", 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54220886999999995 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.447220+0000", 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8471999999999999e-05 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.475 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:880 3.es0 3:7b3c3586:::smithi09731216-12:head [setxattr _header (57) in=7b] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.595605+0000", 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7888323579999996, 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.85208029500000004, 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 880 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595605+0000", 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595605+0000", 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595622+0000", 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6662000000000001e-05 2024-04-18T21:53:06.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595626+0000", 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3810000000000001e-06 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595636+0000", 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8470000000000008e-06 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595642+0000", 2024-04-18T21:53:06.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1670000000000004e-06 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.902273+0000", 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.306631178 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.902573+0000", 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030009699999999998 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.904062+0000", 2024-04-18T21:53:06.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014887380000000001 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.005681+0000", 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101618795 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.447614+0000", 2024-04-18T21:53:06.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44193364499999999 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.447685+0000", 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0784999999999994e-05 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:881 3.es0 3:7b3c3586:::smithi09731216-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:00.595651+0000", 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 5.7887867279999998, 2024-04-18T21:53:06.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1479085549999999, 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 881 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595651+0000", 2024-04-18T21:53:06.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595651+0000", 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595651+0000", 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2200000000000004e-07 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595654+0000", 2024-04-18T21:53:06.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7180000000000001e-06 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595661+0000", 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2599999999999999e-06 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:00.595667+0000", 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9139999999999996e-06 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.005370+0000", 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40970272099999999 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.005660+0000", 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029043999999999999 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.448273+0000", 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.442613074 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.448373+0000", 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9813000000000001e-05 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:01.743559+0000", 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29518609299999998 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.486 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:06.487 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-18T21:53:06.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-18T21:53:06.489 INFO:tasks.rados.rados.0.smithi097.stdout:376: done (6 left) 2024-04-18T21:53:06.489 INFO:tasks.rados.rados.0.smithi097.stdout:377: done (5 left) 2024-04-18T21:53:06.489 INFO:tasks.rados.rados.0.smithi097.stdout:379: done (4 left) 2024-04-18T21:53:06.489 INFO:tasks.rados.rados.0.smithi097.stdout:380: done (3 left) 2024-04-18T21:53:06.489 INFO:tasks.rados.rados.0.smithi097.stdout:382: done (2 left) 2024-04-18T21:53:06.489 INFO:tasks.rados.rados.0.smithi097.stdout:383: done (1 left) 2024-04-18T21:53:06.490 INFO:tasks.rados.rados.0.smithi097.stdout:384: copy_from oid 17 from oid 20 current snap is 32 2024-04-18T21:53:06.490 INFO:tasks.rados.rados.0.smithi097.stdout:385: snap_remove snap 29 2024-04-18T21:53:06.490 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:53:06.482+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:53:06.589 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:06.586+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:06.657 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:06.657 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:53:06.657 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-04-18T21:53:06.657 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:06.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-18T21:53:06.689 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:06.686+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:06.789 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:06.789 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-04-18T21:53:06.789 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-04-18T21:53:06.790 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-04-18T21:53:06.790 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:06.790 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:06.786+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:06.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-18T21:53:06.890 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:53:06.886+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:53:06.970 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.8s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 80.203155742999996, 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51395767000000003, 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.971 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764359+0000", 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764361+0000", 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.61e-06 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764368+0000", 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1600000000000001e-06 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764389+0000", 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1073e-05 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.764409+0000", 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0006000000000001e-05 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866089+0000", 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.101679558 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.866221+0000", 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132022 2024-04-18T21:53:06.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278264+0000", 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.41204366100000001 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278317+0000", 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2580000000000001e-05 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.975 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:103 3.18s0 3:1b279e7b:::smithi09731216-35:head [append 0~573440,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 79.986497279000005, 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6106588159999999, 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 103 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981018+0000", 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981021+0000", 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5470000000000001e-06 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981778+0000", 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00075675099999999995 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981811+0000", 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3200000000000001e-05 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981832+0000", 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0703000000000002e-05 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.277997+0000", 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29616472700000002 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278003+0000", 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3269999999999997e-06 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591581+0000", 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.313578111 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.591677+0000", 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5450000000000003e-05 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.980 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:104 3.18s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 79.985632425000006, 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.714877378, 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981883+0000", 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981885+0000", 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0260000000000002e-06 2024-04-18T21:53:06.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981895+0000", 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7969999999999995e-06 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981918+0000", 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3116e-05 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981931+0000", 2024-04-18T21:53:06.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2850000000000001e-05 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278086+0000", 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29615502199999999 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278088+0000", 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6230000000000001e-06 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696611+0000", 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418522515 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696760+0000", 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000149429 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:105 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:06.985 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 79.985568119000007, 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9142408849999999, 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981947+0000", 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981948+0000", 2024-04-18T21:53:06.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4300000000000002e-07 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981952+0000", 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8739999999999997e-06 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981963+0000", 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.133e-05 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:46.981971+0000", 2024-04-18T21:53:06.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2900000000000002e-06 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278161+0000", 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29618945299999999 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:47.278163+0000", 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.734e-06 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.696819+0000", 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.418656006 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.697058+0000", 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023910700000000001 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:48.896188+0000", 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.19913024800000001 2024-04-18T21:53:06.990 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:318 3.19s0 3:9da3af1e:::smithi09731216-23:head [delete] snapc 5=[2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 68.774642297, 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60663671399999997, 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.991 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 318 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192873+0000", 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192874+0000", 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2699999999999998e-07 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192878+0000", 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0239999999999999e-06 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192886+0000", 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2549999999999997e-06 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.192890+0000", 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.357e-06 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372031+0000", 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.179140625 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372092+0000", 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1457999999999993e-05 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.372626+0000", 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053349599999999995 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.374211+0000", 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001584752 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799295+0000", 2024-04-18T21:53:06.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42508415500000002 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:58.799510+0000", 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021466500000000001 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:329 3.18s0 3:1b279e7b:::smithi09731216-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:53:06.996 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 67.743867023999996, 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60620628200000004, 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 329 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:53:06.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223648+0000", 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6600000000000003e-07 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223651+0000", 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5919999999999999e-06 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223655+0000", 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9890000000000003e-06 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.223658+0000", 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2600000000000001e-06 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:06.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514422+0000", 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.29076426599999999 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.514584+0000", 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000161275 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625434+0000", 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.110850424 2024-04-18T21:53:07.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.625669+0000", 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234315 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:51:59.829854+0000", 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.20418589500000001 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.001 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:360 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~475136] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 63.603328873999999, 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98249101000000005, 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 360 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364186+0000", 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364189+0000", 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8420000000000001e-06 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364486+0000", 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029724500000000002 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364517+0000", 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0994000000000002e-05 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364530+0000", 2024-04-18T21:53:07.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2330999999999999e-05 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364644+0000", 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114673 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.364657+0000", 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2258e-05 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.394826+0000", 2024-04-18T21:53:07.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.030169702 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395495+0000", 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00066827599999999998 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.821805+0000", 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.42631051199999997 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.822025+0000", 2024-04-18T21:53:07.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000220084 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.824602+0000", 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025771510000000002 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944319+0000", 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11971646900000001 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346144+0000", 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.401824764 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346677+0000", 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053370900000000005 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:361 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~622592 in=622592b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:53:07.008 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 63.598588642999999, 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2802270259999999, 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 361 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368926+0000", 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.368928+0000", 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6559999999999999e-06 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369217+0000", 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028880699999999997 2024-04-18T21:53:07.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369238+0000", 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0914999999999999e-05 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369247+0000", 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1910000000000006e-06 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369333+0000", 2024-04-18T21:53:07.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5482000000000006e-05 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369345+0000", 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2147000000000001e-05 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395635+0000", 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026290322000000001 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395761+0000", 2024-04-18T21:53:07.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000126348 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837828+0000", 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.44206649199999998 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837999+0000", 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017102900000000001 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.840298+0000", 2024-04-18T21:53:07.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022987670000000002 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944332+0000", 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10403438199999999 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648919+0000", 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.70458679599999996 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649154+0000", 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000234692 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.017 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:362 3.11s0 3:8fad7925:::smithi09731216-43:head [append 0~147456 in=147456b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 63.597609017000003, 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.279510908, 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.018 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 362 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369906+0000", 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369907+0000", 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0529999999999999e-06 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.369994+0000", 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6688999999999996e-05 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370003+0000", 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3789999999999995e-06 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370010+0000", 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9229999999999996e-06 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370098+0000", 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8146000000000006e-05 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370122+0000", 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3329e-05 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395809+0000", 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.025687331000000001 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.395945+0000", 2024-04-18T21:53:07.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013612499999999999 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941837+0000", 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54589167500000002 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.941940+0000", 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103435 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.942633+0000", 2024-04-18T21:53:07.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069312599999999996 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244973+0000", 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30233927599999999 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649234+0000", 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40426141100000001 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.649417+0000", 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018301 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:363 3.11s0 3:8fad7925:::smithi09731216-43:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 63.597492842999998, 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.380372242, 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.025 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 363 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370022+0000", 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370023+0000", 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.75e-07 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370027+0000", 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8489999999999999e-06 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370039+0000", 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2013e-05 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370043+0000", 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6630000000000001e-06 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370169+0000", 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125506 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.370180+0000", 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0798000000000001e-05 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396197+0000", 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026017723999999999 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.396320+0000", 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122905 2024-04-18T21:53:07.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943149+0000", 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.54682909400000002 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943272+0000", 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012215699999999999 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.943899+0000", 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000627106 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.244975+0000", 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.30107650600000002 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.749791+0000", 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50481555700000003 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.750395+0000", 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00060378899999999998 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:365 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~491520] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 63.59388311, 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97351366699999997, 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.032 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 365 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373632+0000", 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373634+0000", 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.024e-06 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373873+0000", 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000238459 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373899+0000", 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6461999999999998e-05 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.373912+0000", 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2823000000000001e-05 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374028+0000", 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115758 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.374050+0000", 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2314e-05 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.601683+0000", 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22763349599999999 2024-04-18T21:53:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.602117+0000", 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043313999999999999 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828190+0000", 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226073089 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.828481+0000", 2024-04-18T21:53:07.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029162099999999999 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.830297+0000", 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018160229999999999 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944323+0000", 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114025979 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.346866+0000", 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40254325000000002 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.347146+0000", 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027922899999999998 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:366 3.11s0 3:8af06b78:::smithi09731216-10:head [append 0~770048] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:53:07.039 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 63.588264178999999, 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.067907554, 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 366 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379251+0000", 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379253+0000", 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9609999999999999e-06 2024-04-18T21:53:07.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379714+0000", 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00046141699999999999 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379730+0000", 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5585000000000001e-05 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379738+0000", 2024-04-18T21:53:07.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7190000000000001e-06 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379830+0000", 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2551000000000002e-05 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379852+0000", 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.143e-05 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.604888+0000", 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.225036559 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.605008+0000", 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119936 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831601+0000", 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.226592508 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.831778+0000", 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017783799999999999 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.834089+0000", 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023105930000000001 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944325+0000", 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11023567500000001 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.446970+0000", 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50264567000000004 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.447159+0000", 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018811199999999999 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:367 3.11s0 3:8af06b78:::smithi09731216-10:head [setxattr _header (56) in=63b] snapc 9=[9,8,7,6,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:07.046 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 63.587759261999999, 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1678633460000001, 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 367 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379756+0000", 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4899999999999995e-07 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379760+0000", 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4170000000000001e-06 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379766+0000", 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7459999999999997e-06 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379769+0000", 2024-04-18T21:53:07.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.608e-06 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379929+0000", 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015980200000000001 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379953+0000", 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3764000000000002e-05 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707668+0000", 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32771509500000001 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707855+0000", 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186987 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835388+0000", 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127533496 2024-04-18T21:53:07.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.835835+0000", 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00044705299999999999 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.836767+0000", 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00093163299999999996 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.944330+0000", 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107563336 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547378+0000", 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60304798000000004 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.547619+0000", 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024088000000000001 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:368 3.11s0 3:8af06b78:::smithi09731216-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-18T21:53:07.053 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 63.587723332000003, 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6760458580000002, 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 368 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379792+0000", 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.91e-07 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379799+0000", 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2690000000000002e-06 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379805+0000", 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7989999999999999e-06 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379809+0000", 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4929999999999998e-06 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.379996+0000", 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000186959 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.380015+0000", 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9837e-05 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.707952+0000", 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.32793653499999997 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.708113+0000", 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160536 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837592+0000", 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12947945499999999 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:03.837760+0000", 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016796899999999999 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.647833+0000", 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.81007341499999996 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:04.648038+0000", 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020446900000000001 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:05.055838+0000", 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.407799831 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:737 3.8s0 3:1b279e7b:::smithi09731216-35:head [append 0~147456] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:50.958634+0000", 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.008881244000001, 2024-04-18T21:53:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60400940000000003, 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 737 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958634+0000", 2024-04-18T21:53:07.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958634+0000", 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958635+0000", 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0070000000000001e-06 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958714+0000", 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9270999999999994e-05 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958725+0000", 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0458999999999999e-05 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958732+0000", 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2559999999999996e-06 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.063 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.055021+0000", 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.096288755000000004 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.055129+0000", 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000108717 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.055852+0000", 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00072295599999999997 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.056907+0000", 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001055159 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.562580+0000", 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.50567232699999998 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.562643+0000", 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3492999999999997e-05 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.065 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:738 3.8s0 3:1b279e7b:::smithi09731216-35:head [setxattr _header (57) in=7b] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:50.958754+0000", 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 16.008760820999999, 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.60411800999999998, 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.066 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 738 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958754+0000", 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958754+0000", 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958755+0000", 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.37e-07 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958758+0000", 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9679999999999998e-06 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:50.958763+0000", 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6620000000000002e-06 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.058839+0000", 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10007514200000001 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.159007+0000", 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10016875 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.159108+0000", 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100278 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.159609+0000", 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00050155700000000002 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.160564+0000", 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00095452899999999997 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.562777+0000", 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.40221319900000002 2024-04-18T21:53:07.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:51.562872+0000", 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5388000000000002e-05 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:813 3.9s0 3:9941efaf:::smithi09731216-5:head [copy-from ver 13] snapc 19=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:52:55.049609+0000", 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.917905888, 2024-04-18T21:53:07.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.53574875499999997, 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 813 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049609+0000", 2024-04-18T21:53:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049609+0000", 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049612+0000", 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8159999999999998e-06 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049626+0000", 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3903000000000001e-05 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049672+0000", 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6217000000000002e-05 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.049690+0000", 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7479999999999999e-05 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.051766+0000", 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020764519999999999 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.051939+0000", 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000173246 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.294069+0000", 2024-04-18T21:53:07.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24212983299999999 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.361592+0000", 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.067522847999999996 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.584633+0000", 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22304118100000001 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:52:55.585358+0000", 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00072477900000000005 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:916 3.8s0 3:1b279e7b:::smithi09731216-35:head [append 0~802816] snapc 1d=[1d,1c,1b] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:02.457439+0000", 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.5100764240000002, 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62409359200000003, 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.077 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 916 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.457439+0000", 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.457439+0000", 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.457442+0000", 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3900000000000002e-06 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.458350+0000", 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00090784700000000004 2024-04-18T21:53:07.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.458387+0000", 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7169999999999998e-05 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.458414+0000", 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6769999999999999e-05 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.458570+0000", 2024-04-18T21:53:07.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155725 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.458584+0000", 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4212999999999999e-05 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.565933+0000", 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107349423 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.566101+0000", 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016759199999999999 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.570366+0000", 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004265385 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.582024+0000", 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011658043999999999 2024-04-18T21:53:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:03.081365+0000", 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49934119500000002 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:03.081532+0000", 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016683799999999999 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:917 3.8s0 3:1b279e7b:::smithi09731216-35:head [append 0~425984] snapc 1d=[1d,1c,1b] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:02.459822+0000", 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.5076935049999998, 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.62218198400000002, 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 917 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.459822+0000", 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.459822+0000", 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.459824+0000", 2024-04-18T21:53:07.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9829999999999999e-06 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.460295+0000", 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00047095500000000001 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.460325+0000", 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0883000000000002e-05 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.460347+0000", 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1633000000000001e-05 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.460514+0000", 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016701500000000001 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.460528+0000", 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.438e-05 2024-04-18T21:53:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.573855+0000", 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.113326657 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.574034+0000", 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000178438 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.576290+0000", 2024-04-18T21:53:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022562599999999999 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.588006+0000", 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011715929 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:03.081876+0000", 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.49387042599999997 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:03.082004+0000", 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012742499999999999 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.4509.0:924 3.11s0 3:8af06b78:::smithi09731216-10:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106)", 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-04-18T21:53:02.466615+0000", 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4.5009000989999999, 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.51131596599999996, 2024-04-18T21:53:07.090 INFO:teuthology.orchestra.run.smithi097.stdout: "continuous": false, 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.4509", 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 924 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.466615+0000", 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.466615+0000", 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.466616+0000", 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7899999999999999e-07 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-04-18T21:53:07.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.466618+0000", 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8040000000000002e-06 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.466624+0000", 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9209999999999997e-06 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.466629+0000", 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0579999999999997e-06 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.482359+0000", 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015729781000000002 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.482380+0000", 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1019999999999999e-05 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.585284+0000", 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102903836 2024-04-18T21:53:07.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.585455+0000", 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017119800000000001 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-04-18T21:53:02.977931+0000", 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.39247586899999998 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-18T21:53:07.095 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:07.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-18T21:53:07.098 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:07.094+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:07.197 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:07.198+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:07.227 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 110/93 ECSubWriteReply(tid=396, last_complete=110'216, committed=1, applied=1))", 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.087310+0000", 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.14030964700000001, 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14034661900000001, 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.087310+0000", 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.087310+0000", 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.087311+0000", 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2270000000000001e-06 2024-04-18T21:53:07.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.087318+0000", 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3560000000000004e-06 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.087341+0000", 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.243e-05 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.087353+0000", 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2442999999999999e-05 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 110/93 ECSubWriteReply(tid=396, last_complete=110'216, committed=1, applied=1))", 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.186522+0000", 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.041097717999999998, 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.041222222000000003, 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.231 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186522+0000", 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186522+0000", 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186524+0000", 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3439999999999999e-06 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186537+0000", 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2833000000000001e-05 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186569+0000", 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1866000000000003e-05 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186588+0000", 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8819999999999999e-05 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 110/93 ECSubWriteReply(tid=396, last_complete=110'216, committed=1, applied=1))", 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.186925+0000", 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.040693891000000003, 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.040893539999999999, 2024-04-18T21:53:07.234 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186925+0000", 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186925+0000", 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186927+0000", 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3999999999999999e-06 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186936+0000", 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4159999999999993e-06 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186959+0000", 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2659000000000001e-05 2024-04-18T21:53:07.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.186972+0000", 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2629999999999999e-05 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 3 2024-04-18T21:53:07.237 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:07.241 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-18T21:53:07.299 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:53:07.294+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:53:07.359 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:07.359 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:07.359 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:53:07.359 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:53:07.360 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:07.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-18T21:53:07.417 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:53:07.414+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:53:07.459 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is set 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:385: done (2 left) 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:386: rollback oid 32 current snap is 32 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 32 to 26 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:381: finishing write tid 7 to smithi09731216-13 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:381: oid 13 updating version 0 to 36 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:381: oid 13 updating version 36 to 37 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:381: oid 13 version 37 is already newer than 35 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:381: oid 13 updating version 37 to 40 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:381: oid 13 version 40 is already newer than 38 2024-04-18T21:53:07.465 INFO:tasks.rados.rados.0.smithi097.stdout:381: oid 13 version 40 is already newer than 39 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 13 v 40 (ObjNum 143 snap 31 seq_num 143) dirty exists 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:381: left oid 13 (ObjNum 143 snap 31 seq_num 143) 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:384: finishing copy_from racing read to smithi09731216-17 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:384: finishing copy_from to smithi09731216-17 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 17 v 181 (ObjNum 126 snap 24 seq_num 126) dirty exists 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:381: done (2 left) 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:384: done (1 left) 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:387: read oid 28 snap -1 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:387: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 21 current snap is 32 2024-04-18T21:53:07.466 INFO:tasks.rados.rados.0.smithi097.stdout:388: seq_num 144 ranges {2277376=16384} 2024-04-18T21:53:07.484 INFO:tasks.rados.rados.0.smithi097.stdout:388: writing smithi09731216-21 from 2277376 to 2293760 tid 1 2024-04-18T21:53:07.484 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 4 current snap is 32 2024-04-18T21:53:07.484 INFO:tasks.rados.rados.0.smithi097.stdout:389: seq_num 145 ranges {65536=32768} 2024-04-18T21:53:07.485 INFO:tasks.rados.rados.0.smithi097.stdout:389: writing smithi09731216-4 from 65536 to 98304 tid 1 2024-04-18T21:53:07.485 INFO:tasks.rados.rados.0.smithi097.stdout:390: delete oid 9 current snap is 32 2024-04-18T21:53:07.533 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:07.530+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:07.547 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:07.547 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:53:07.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:53:07.547 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:07.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.547 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.290660461000002, 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82320560300000001, 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254834+0000", 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.254837+0000", 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9000000000000002e-06 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255465+0000", 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000628435 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255498+0000", 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2305999999999999e-05 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255523+0000", 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5137e-05 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078012+0000", 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82248975599999996 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078039+0000", 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7069000000000001e-05 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.551 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.3s0 3.fef99513 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.289933555999994, 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82259326700000002, 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.552 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255561+0000", 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255563+0000", 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.72e-06 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255574+0000", 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0993e-05 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255598+0000", 2024-04-18T21:53:07.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4329999999999999e-05 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255615+0000", 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7569000000000001e-05 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078131+0000", 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82251584300000002 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078154+0000", 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2812000000000001e-05 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:93 3.3s0 3.fef99513 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.289845614000001, 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822627518, 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.556 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 93 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255649+0000", 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255650+0000", 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2330000000000001e-06 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255657+0000", 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5000000000000002e-06 2024-04-18T21:53:07.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255681+0000", 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3354e-05 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.255705+0000", 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3932999999999999e-05 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078240+0000", 2024-04-18T21:53:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822535609 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.078276+0000", 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5889000000000001e-05 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:73 3.13s0 3:c8da3643:::smithi09731216-25:head [append 0~409600,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:53:07.560 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 80.348762144000005, 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.609707234, 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 73 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:53:07.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196732+0000", 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196734+0000", 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829999999999999e-06 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196977+0000", 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024357200000000001 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.196998+0000", 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0503e-05 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197013+0000", 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5536000000000002e-05 2024-04-18T21:53:07.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197060+0000", 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6519e-05 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197061+0000", 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2900000000000002e-07 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591261+0000", 2024-04-18T21:53:07.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3942002449999999 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.591644+0000", 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038315000000000002 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.593905+0000", 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022609050000000001 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697736+0000", 2024-04-18T21:53:07.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10383110700000001 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806236+0000", 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10850008999999999 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.806439+0000", 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020309500000000001 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.566 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:74 3.13s0 3:c8da3643:::smithi09731216-25:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 80.348457800000006, 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7102205860000002, 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 74 2024-04-18T21:53:07.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197037+0000", 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197038+0000", 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9000000000000005e-07 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197043+0000", 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1100000000000002e-06 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197053+0000", 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0267e-05 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197066+0000", 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.327e-05 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197104+0000", 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8039e-05 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197105+0000", 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8400000000000004e-07 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595282+0000", 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3981771869999999 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.595543+0000", 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026070899999999997 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.596517+0000", 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00097469500000000003 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.697956+0000", 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10143811899999999 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907075+0000", 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20911996799999999 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.907257+0000", 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018164799999999999 2024-04-18T21:53:07.572 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:91 3.13s0 3:c8a99f7f:::smithi09731216-31:head [append 0~507904,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 80.347931736999996, 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.812483013, 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 91 2024-04-18T21:53:07.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197563+0000", 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8599999999999998e-07 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197879+0000", 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000315495 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197893+0000", 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4143e-05 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197908+0000", 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4932999999999999e-05 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197987+0000", 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9288000000000004e-05 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197989+0000", 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.137e-06 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.698727+0000", 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.500737242 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.699015+0000", 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028849499999999999 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.700958+0000", 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019429969999999999 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798717+0000", 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097759288999999999 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.009932+0000", 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.211214651 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010046+0000", 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113657 2024-04-18T21:53:07.578 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:92 3.13s0 3:c8a99f7f:::smithi09731216-31:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 80.347569895999996, 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8124102710000001, 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.579 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 92 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197924+0000", 2024-04-18T21:53:07.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197925+0000", 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000005e-07 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197931+0000", 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6799999999999998e-06 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197943+0000", 2024-04-18T21:53:07.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1620000000000001e-05 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.197953+0000", 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-05 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7586000000000001e-05 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.198011+0000", 2024-04-18T21:53:07.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5599999999999995e-07 2024-04-18T21:53:07.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702431+0000", 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.504419629 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.702670+0000", 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023948700000000001 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.703977+0000", 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00130674 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.798719+0000", 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094742397000000006 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010219+0000", 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21149920999999999 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.010335+0000", 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011602 2024-04-18T21:53:07.587 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:234 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.511677767999998, 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.87684132699999995, 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.588 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 234 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:53:07.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033817+0000", 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9599999999999998e-07 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033821+0000", 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.653e-06 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033834+0000", 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3303e-05 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.033847+0000", 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2327000000000001e-05 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808105+0000", 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.77425798199999996 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.808280+0000", 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017508500000000001 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:53.910658+0000", 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102378081 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:307 3.3s0 3:c170a397:::smithi09731216-3:head [read 0~1097728,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:53:07.592 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 69.459111406000005, 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81219510800000005, 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 307 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:53:07.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086383+0000", 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086384+0000", 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9699999999999995e-07 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086388+0000", 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3900000000000003e-06 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086398+0000", 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8369999999999992e-06 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086406+0000", 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3599999999999996e-06 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086727+0000", 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032101900000000002 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.086748+0000", 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0917e-05 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292674+0000", 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20592602600000001 2024-04-18T21:53:07.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.292826+0000", 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015169700000000001 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.898578+0000", 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60575206500000001 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:325 3.3s0 3:c26b2dda:::smithi09731216-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 68.319320520000005, 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.79501891000000002, 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 325 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226174+0000", 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4400000000000001e-07 2024-04-18T21:53:07.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226176+0000", 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1100000000000001e-06 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226182+0000", 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5230000000000003e-06 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.226186+0000", 2024-04-18T21:53:07.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0180000000000003e-06 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.230252+0000", 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040660369999999998 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.230395+0000", 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014306400000000001 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.817971+0000", 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58757587300000003 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.818220+0000", 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024919800000000001 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:00.021193+0000", 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20297274300000001 2024-04-18T21:53:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:533 3.3s0 3:c2fd059f:::smithi09731216-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.035228967000002, 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.85709232000000002, 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.603 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 533 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510265+0000", 2024-04-18T21:53:07.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510266+0000", 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-06 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510271+0000", 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7670000000000003e-06 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.510283+0000", 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1965e-05 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.610415+0000", 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100132151 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.927765+0000", 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31734968699999999 2024-04-18T21:53:07.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:24.927918+0000", 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000152702 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133621+0000", 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.205703102 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.133704+0000", 2024-04-18T21:53:07.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3040000000000005e-05 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.367358+0000", 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23365385799999999 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:608 3.1s1 3:809f338e:::smithi09731216-11:head [copy-from ver 112] snapc 14=[10,a,6] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e78)", 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.166366685, 2024-04-18T21:53:07.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.881166288, 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 608 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379128+0000", 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379131+0000", 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1e-06 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379144+0000", 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2904e-05 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379188+0000", 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3909000000000003e-05 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379214+0000", 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6610000000000001e-05 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.675749+0000", 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29653525600000002 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.675755+0000", 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0129999999999999e-06 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.901288+0000", 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.225532912 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.901574+0000", 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028603400000000001 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.146507+0000", 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.244932181 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.150976+0000", 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044695780000000001 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.259562+0000", 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10858633299999999 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.260294+0000", 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00073145800000000004 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.614 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:609 3.1s1 3:809f338e:::smithi09731216-11:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.166240326, 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7716583269999999, 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 609 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:53:07.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379254+0000", 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379255+0000", 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.474e-06 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379263+0000", 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7530000000000001e-06 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379287+0000", 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3416000000000001e-05 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.379305+0000", 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.819e-05 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.775808+0000", 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39650268500000002 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:34.775813+0000", 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8329999999999999e-06 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.902311+0000", 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.126497995 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:35.902394+0000", 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2752999999999999e-05 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.149564+0000", 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.247170267 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.150153+0000", 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0005886 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:36.150912+0000", 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00075936099999999996 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:680 3.3s0 3:c0b736af:::smithi09731216-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:45.560822+0000", 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.984672240999998, 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.83707749099999995, 2024-04-18T21:53:07.620 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 680 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560822+0000", 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560822+0000", 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560823+0000", 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8700000000000004e-07 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560827+0000", 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7359999999999999e-06 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560837+0000", 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0349999999999999e-05 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.560846+0000", 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5099999999999998e-06 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.591970+0000", 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031124037 2024-04-18T21:53:07.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:45.592184+0000", 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021478400000000001 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.177742+0000", 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.58555721100000002 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.178151+0000", 2024-04-18T21:53:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000409178 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.397900+0000", 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.219748798 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:727 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~0] snapc 18=[17,10] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:49.474328+0000", 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.071166705, 2024-04-18T21:53:07.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1869300279999999, 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 727 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474328+0000", 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474328+0000", 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474331+0000", 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.943e-06 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474344+0000", 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.31e-05 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474376+0000", 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.222e-05 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474401+0000", 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4902999999999999e-05 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167528+0000", 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69312696200000001 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167532+0000", 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5329999999999996e-06 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.111274+0000", 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94374193200000001 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.111739+0000", 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046505500000000003 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.113892+0000", 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021522770000000002 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.321845+0000", 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.207953264 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.661063+0000", 2024-04-18T21:53:07.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.33921768099999999 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.661258+0000", 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019515799999999999 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:728 3.3s0 3:c2fd059f:::smithi09731216-20:head [rollback 17] snapc 18=[17,10] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:49.474489+0000", 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.071005323000001, 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3914702019999998, 2024-04-18T21:53:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 728 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474489+0000", 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474489+0000", 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474490+0000", 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.063e-06 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474496+0000", 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9370000000000001e-06 2024-04-18T21:53:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474526+0000", 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0389e-05 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474542+0000", 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5183e-05 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167563+0000", 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.693021675 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167564+0000", 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6000000000000003e-07 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.218010+0000", 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0504465030000001 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.218206+0000", 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000196021 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.219222+0000", 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010155660000000001 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.322206+0000", 2024-04-18T21:53:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10298433999999999 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.865666+0000", 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.54345940500000001 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.865959+0000", 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029346 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:729 3.3s0 3:c2fd059f:::smithi09731216-20:head [append 0~0] snapc 18=[17,10] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:49.474558+0000", 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.070935924, 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.392365496, 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 729 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474558+0000", 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474558+0000", 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474559+0000", 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9299999999999997e-07 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474563+0000", 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6899999999999998e-06 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474574+0000", 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.08e-05 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.474582+0000", 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4100000000000008e-06 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167607+0000", 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69302461500000001 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.167609+0000", 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5739999999999998e-06 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.325225+0000", 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.157615944 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.325417+0000", 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019197800000000001 2024-04-18T21:53:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.326488+0000", 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010710439999999999 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.328451+0000", 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019627529999999998 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.866764+0000", 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.53831266099999997 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.866924+0000", 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016023400000000001 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:771 3.3s0 3:c170a397:::smithi09731216-3:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.055730+0000", 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.489764214000001, 2024-04-18T21:53:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81168786100000001, 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 771 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055730+0000", 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055730+0000", 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055734+0000", 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4800000000000001e-06 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055748+0000", 2024-04-18T21:53:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.413e-05 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055779+0000", 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0854e-05 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.055801+0000", 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2656000000000001e-05 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.326954+0000", 2024-04-18T21:53:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.27115272000000001 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.327187+0000", 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023294099999999999 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.867418+0000", 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.54023107999999997 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:821 3.3s0 3:c26b2dda:::smithi09731216-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:57.067499+0000", 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.477995052000001, 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.760915016, 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 821 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067499+0000", 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067499+0000", 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067500+0000", 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7400000000000002e-07 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067504+0000", 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4060000000000001e-06 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.067517+0000", 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3356000000000001e-05 2024-04-18T21:53:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.167607+0000", 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100089574 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.176136+0000", 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0085291789999999996 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.176279+0000", 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143793 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.722494+0000", 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.54621423999999996 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.722784+0000", 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029031700000000001 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.828414+0000", 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10563027699999999 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:887 3.3s0 3:c26b2dda:::smithi09731216-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.536332+0000", 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.0091619400000003, 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90024217299999998, 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 887 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536332+0000", 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536332+0000", 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536334+0000", 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3239999999999999e-06 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536339+0000", 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7259999999999999e-06 2024-04-18T21:53:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536363+0000", 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.357e-05 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.536379+0000", 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6288999999999999e-05 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:01.032418+0000", 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49603846000000001 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:01.032485+0000", 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7457000000000001e-05 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:01.234409+0000", 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20192394299999999 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:01.234696+0000", 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028700699999999998 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:01.436575+0000", 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20187839699999999 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.659 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:07.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-18T21:53:07.662 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:53:07.658+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:53:07.666 INFO:tasks.rados.rados.0.smithi097.stdout:390: done (4 left) 2024-04-18T21:53:07.666 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 17 current snap is 32 2024-04-18T21:53:07.666 INFO:tasks.rados.rados.0.smithi097.stdout:391: seq_num 147 ranges {2129920=65536} 2024-04-18T21:53:07.693 INFO:tasks.rados.rados.0.smithi097.stdout:391: writing smithi09731216-17 from 2129920 to 2195456 tid 1 2024-04-18T21:53:07.693 INFO:tasks.rados.rados.0.smithi097.stdout:392: delete oid 34 current snap is 32 2024-04-18T21:53:07.720 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-18T21:53:07.720 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-18T21:53:07.720 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-18T21:53:07.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-18T21:53:07.765 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:53:07.762+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:53:07.838 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:07.838 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:07.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.838 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 111/95 ECSubWriteReply(tid=246, last_complete=111'19, committed=1, applied=1))", 2024-04-18T21:53:07.838 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.773095+0000", 2024-04-18T21:53:07.838 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.064332058999999997, 2024-04-18T21:53:07.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064388250999999994, 2024-04-18T21:53:07.838 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773095+0000", 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773095+0000", 2024-04-18T21:53:07.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773095+0000", 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.63e-07 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773097+0000", 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.967e-06 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773106+0000", 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8210000000000008e-06 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773111+0000", 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.079e-06 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.10s2 111/95 ECSubWrite(tid=164, reqid=osd.6.0:164, at_version=111'77, trim_to=0'0, roll_forward_to=109'75))", 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.773167+0000", 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.064259471999999998, 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064446375, 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773167+0000", 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773167+0000", 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773167+0000", 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2499999999999999e-07 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773189+0000", 2024-04-18T21:53:07.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2076e-05 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773200+0000", 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0249e-05 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773208+0000", 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8620000000000003e-06 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 111/95 ECSubWriteReply(tid=246, last_complete=111'19, committed=1, applied=1))", 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.773221+0000", 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.064206089999999993, 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064469995000000002, 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773221+0000", 2024-04-18T21:53:07.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773221+0000", 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773221+0000", 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7899999999999999e-07 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773224+0000", 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8509999999999999e-06 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773231+0000", 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6539999999999999e-06 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773235+0000", 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7929999999999999e-06 2024-04-18T21:53:07.847 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 111/95 ECSubWriteReply(tid=246, last_complete=111'19, committed=1, applied=1))", 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.773296+0000", 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.064130954000000004, 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064469293999999996, 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773296+0000", 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773296+0000", 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773296+0000", 2024-04-18T21:53:07.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5600000000000001e-07 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773299+0000", 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1300000000000001e-06 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773318+0000", 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8589000000000001e-05 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773325+0000", 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0129999999999996e-06 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 111/95 ECSubWriteReply(tid=246, last_complete=111'19, committed=1, applied=1))", 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.773302+0000", 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.064124733000000003, 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064535219000000005, 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773302+0000", 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773302+0000", 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773304+0000", 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.277e-06 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773332+0000", 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7529e-05 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773359+0000", 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7753e-05 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773379+0000", 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9796000000000002e-05 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.11s5 111/97 ECSubWrite(tid=220, reqid=osd.3.0:220, at_version=111'40, trim_to=0'0, roll_forward_to=109'38))", 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.773336+0000", 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.064091061000000005, 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064577311999999998, 2024-04-18T21:53:07.854 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773336+0000", 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773336+0000", 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773336+0000", 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7199999999999999e-07 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773362+0000", 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5434000000000001e-05 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773372+0000", 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0932e-05 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773381+0000", 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4959999999999996e-06 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:978 3.cs0 3:348e44d8:::smithi09731216-34:head [delete] snapc 21=[21,20,1e,1d,1c,1b] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.773418+0000", 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.064009112000000007, 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064583270999999998, 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.857 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 978 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773418+0000", 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773418+0000", 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773419+0000", 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.708e-06 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773424+0000", 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5600000000000004e-06 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773439+0000", 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4779e-05 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773448+0000", 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8239999999999995e-06 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773666+0000", 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021729099999999999 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.773792+0000", 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012662099999999999 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s5 111/93 ECSubWrite(tid=398, reqid=osd.4.0:397, at_version=0'0, trim_to=0'0, roll_forward_to=111'218))", 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:07.767977+0000", 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.069450068000000004, 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.070134754999999993, 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:07.861 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "no flag points reached", 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.767977+0000", 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.767977+0000", 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:07.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.767978+0000", 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.632e-06 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.767996+0000", 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.774e-05 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:07.768022+0000", 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5871999999999999e-05 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:07.863 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:07.864 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:53:07.864 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 8 2024-04-18T21:53:07.864 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:07.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-18T21:53:07.865 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-04-18T21:53:07.862+0000 7f4777d6f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25095) UID: 0 2024-04-18T21:53:07.966 INFO:tasks.ceph.osd.6.smithi104.stderr:2024-04-18T21:53:07.962+0000 7f3db0115640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25097) UID: 0 2024-04-18T21:53:07.977 INFO:tasks.rados.rados.0.smithi097.stdout:389: finishing write tid 1 to smithi09731216-4 2024-04-18T21:53:07.985 INFO:tasks.rados.rados.0.smithi097.stdout:386: finishing rollback tid 0 to smithi09731216-32 2024-04-18T21:53:07.985 INFO:tasks.rados.rados.0.smithi097.stdout:386: finishing rollback tid 1 to smithi09731216-32 2024-04-18T21:53:07.985 INFO:tasks.rados.rados.0.smithi097.stdout:386: finishing rollback tid 2 to smithi09731216-32 2024-04-18T21:53:07.985 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 32 v 72 (ObjNum 122 snap 24 seq_num 122) dirty exists 2024-04-18T21:53:08.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-18T21:53:08.032 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:08.032 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:08.033 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:53:08.033 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:53:08.033 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:08.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-18T21:53:08.073 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-04-18T21:53:08.070+0000 7fe59fdce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25113) UID: 0 2024-04-18T21:53:08.075 INFO:tasks.rados.rados.0.smithi097.stdout:391: finishing write tid 1 to smithi09731216-17 2024-04-18T21:53:08.075 INFO:tasks.rados.rados.0.smithi097.stdout:386: done (5 left) 2024-04-18T21:53:08.075 INFO:tasks.rados.rados.0.smithi097.stdout:392: done (4 left) 2024-04-18T21:53:08.075 INFO:tasks.rados.rados.0.smithi097.stdout:append oid 30 current snap is 32 2024-04-18T21:53:08.075 INFO:tasks.rados.rados.0.smithi097.stdout:393: seq_num 149 ranges {1277952=98304} 2024-04-18T21:53:08.086 INFO:tasks.rados.rados.0.smithi097.stdout:393: writing smithi09731216-30 from 1277952 to 1376256 tid 1 2024-04-18T21:53:08.086 INFO:tasks.rados.rados.0.smithi097.stdout:394: copy_from oid 2 from oid 29 current snap is 32 2024-04-18T21:53:08.086 INFO:tasks.rados.rados.0.smithi097.stdout:395: setattr oid 13 current snap is 32 2024-04-18T21:53:08.086 INFO:tasks.rados.rados.0.smithi097.stdout:388: finishing write tid 1 to smithi09731216-21 2024-04-18T21:53:08.090 INFO:tasks.rados.rados.0.smithi097.stdout:396: read oid 9 snap -1 2024-04-18T21:53:08.091 INFO:tasks.rados.rados.0.smithi097.stdout:396: expect deleted 2024-04-18T21:53:08.091 INFO:tasks.rados.rados.0.smithi097.stdout:397: snap_create 2024-04-18T21:53:08.168 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:53:08.162+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:53:08.175 INFO:tasks.rados.rados.0.smithi097.stdout:391: finishing write tid 2 to smithi09731216-17 2024-04-18T21:53:08.178 INFO:tasks.rados.rados.0.smithi097.stdout:389: finishing write tid 2 to smithi09731216-4 2024-04-18T21:53:08.187 INFO:tasks.rados.rados.0.smithi097.stdout:388: finishing write tid 2 to smithi09731216-21 2024-04-18T21:53:08.209 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:08.209 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:53:08.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:53:08.209 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:08.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:100 3.cs0 3:348e44d8:::smithi09731216-34:head [append 0~557056,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.547949286000005, 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50652407300000002, 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100 2024-04-18T21:53:08.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659952+0000", 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.659953+0000", 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.821e-06 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660381+0000", 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042725800000000002 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660395+0000", 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.394e-05 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660404+0000", 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6520000000000001e-06 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760564+0000", 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10015995900000001 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.760699+0000", 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013526 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762105+0000", 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00140595 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965039+0000", 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20293309400000001 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166421+0000", 2024-04-18T21:53:08.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.201382373 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.166476+0000", 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4765999999999997e-05 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:101 3.cs0 3:348e44d8:::smithi09731216-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.547479546000005, 2024-04-18T21:53:08.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60956795200000002, 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660421+0000", 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660422+0000", 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-07 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660426+0000", 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3830000000000002e-06 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660436+0000", 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8679999999999994e-06 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660443+0000", 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2999999999999998e-06 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762872+0000", 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.1024294 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.762973+0000", 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001015 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763506+0000", 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053209199999999996 2024-04-18T21:53:08.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.965047+0000", 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20154100999999999 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269839+0000", 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.304792377 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.269989+0000", 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015033399999999999 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:102 3.cs0 3:348e44d8:::smithi09731216-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.547429996999995, 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.92759091000000005, 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.221 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660471+0000", 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660472+0000", 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0999999999999998e-07 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660476+0000", 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0879999999999997e-06 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660493+0000", 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6963e-05 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.660502+0000", 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9579999999999996e-06 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763793+0000", 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10329121600000001 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.763891+0000", 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8127000000000004e-05 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372039+0000", 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60814838199999999 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.372189+0000", 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014991200000000001 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.588062+0000", 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.215872654 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:78 3.15s0 3:ab33d1b5:::smithi09731216-26:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 81.226884573000007, 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1076682130000002, 2024-04-18T21:53:08.226 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981016+0000", 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981019+0000", 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.447e-06 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981025+0000", 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6810000000000001e-06 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981044+0000", 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8856999999999999e-05 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:46.981056+0000", 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1921e-05 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164029+0000", 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18297308600000001 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:47.164033+0000", 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0369999999999996e-06 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885490+0000", 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7214569900000001 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:48.885771+0000", 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028064300000000002 2024-04-18T21:53:08.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:49.088685+0000", 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.202913551 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:463 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~638976] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.833811724, 2024-04-18T21:53:08.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50349991299999997, 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 463 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374089+0000", 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.374091+0000", 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.84e-06 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375137+0000", 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010459 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375156+0000", 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8978e-05 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375169+0000", 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2660000000000001e-05 2024-04-18T21:53:08.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571128+0000", 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19595929000000001 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.571311+0000", 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018294599999999999 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.572699+0000", 2024-04-18T21:53:08.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001388715 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675602+0000", 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102902721 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877476+0000", 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20187386299999999 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.877589+0000", 2024-04-18T21:53:08.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:464 3.cs0 3:3c73a4f8:::smithi09731216-45:head [append 0~131072] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.832698139000001, 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50427034999999998, 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.237 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 464 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375203+0000", 2024-04-18T21:53:08.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375204+0000", 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3650000000000001e-06 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375344+0000", 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013967800000000001 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375357+0000", 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3580999999999999e-05 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375367+0000", 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0117e-05 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573497+0000", 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198129889 2024-04-18T21:53:08.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.573606+0000", 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010875 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.574480+0000", 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00087374900000000005 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.675605+0000", 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10112497400000001 2024-04-18T21:53:08.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879333+0000", 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20372839500000001 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.879473+0000", 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013985199999999999 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:465 3.cs0 3:3c73a4f8:::smithi09731216-45:head [setxattr _header (57) in=7b] snapc f=[f,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.832515604000001, 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60269858399999998, 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 465 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375385+0000", 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375386+0000", 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6300000000000004e-07 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375391+0000", 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.729e-06 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375401+0000", 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9569999999999997e-06 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375409+0000", 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6620000000000005e-06 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777143+0000", 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40173476200000002 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777239+0000", 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5712999999999997e-05 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777695+0000", 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045595 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.778376+0000", 2024-04-18T21:53:08.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068123800000000003 2024-04-18T21:53:08.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.977952+0000", 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.199575432 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.978084+0000", 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000132278 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:466 3.cs0 3:3c73a4f8:::smithi09731216-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.832480916999998, 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006726617, 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 466 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.251 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375420+0000", 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375421+0000", 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.009e-06 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375424+0000", 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.495e-06 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375435+0000", 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0360000000000001e-05 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.375442+0000", 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6389999999999992e-06 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777929+0000", 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40248657999999998 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:18.777993+0000", 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2024-04-18T21:53:08.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078687+0000", 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30069407399999998 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.078849+0000", 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162065 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:19.382147+0000", 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30329795300000001 2024-04-18T21:53:08.255 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:696 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.730259392000001, 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8198962060000001, 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.256 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 696 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477642+0000", 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8899999999999998e-07 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477645+0000", 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2030000000000001e-06 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477655+0000", 2024-04-18T21:53:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8050000000000001e-06 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.477663+0000", 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2459999999999997e-06 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.554664+0000", 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.077001179000000003 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:46.554666+0000", 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1399999999999998e-06 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.588198+0000", 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0335324410000002 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.588403+0000", 2024-04-18T21:53:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020478799999999999 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.992774+0000", 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.404371174 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:48.993448+0000", 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067422999999999995 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:49.297538+0000", 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30408921100000003 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:744 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~3768320,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.249480537, 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52158899199999997, 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.262 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 744 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958420+0000", 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958421+0000", 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0499999999999999e-06 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958427+0000", 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0059999999999998e-06 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958440+0000", 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2269999999999999e-05 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958453+0000", 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2893e-05 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.961672+0000", 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032194329999999998 2024-04-18T21:53:08.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.962277+0000", 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060548399999999999 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.480009+0000", 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51773185600000005 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:745 3.5s0 3:a2c4e662:::smithi09731216-50:head [read 0~3768320,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.249408600999999, 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62853464000000003, 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 745 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.267 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958492+0000", 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958493+0000", 2024-04-18T21:53:08.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7700000000000003e-07 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958498+0000", 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.096e-06 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958513+0000", 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4904e-05 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:50.958522+0000", 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5760000000000006e-06 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.064677+0000", 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106155732 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.065006+0000", 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032830000000000001 2024-04-18T21:53:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.587027+0000", 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52202115500000001 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:759 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.164425219000002, 2024-04-18T21:53:08.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62612275900000003, 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 759 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:53:08.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043476+0000", 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043479+0000", 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8430000000000001e-06 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043492+0000", 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.322e-05 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043518+0000", 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6567000000000002e-05 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043533+0000", 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4863e-05 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.201592+0000", 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.158059226 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.201907+0000", 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031459700000000001 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.669598+0000", 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46769144299999998 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.275 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:760 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.164334444000001, 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.62795852399999996, 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 760 2024-04-18T21:53:08.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043566+0000", 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043567+0000", 2024-04-18T21:53:08.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6700000000000002e-07 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043573+0000", 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3499999999999996e-06 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043585+0000", 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.205e-05 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043594+0000", 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8300000000000002e-06 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.202744+0000", 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15915027200000001 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.202986+0000", 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000241883 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.671525+0000", 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.468539172 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:761 3.cs0 3:3c73a4f8:::smithi09731216-45:17 [read 0~1425408,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.164294470000002, 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.63223503700000006, 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.280 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 761 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043606+0000", 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043607+0000", 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0299999999999998e-07 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043611+0000", 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3030000000000001e-06 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043621+0000", 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0023999999999999e-05 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.043629+0000", 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8460000000000008e-06 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204276+0000", 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16064708899999999 2024-04-18T21:53:08.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204439+0000", 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016309500000000001 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.675841+0000", 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47140197700000003 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.284 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpReadReply(3.cs0 94/84 ECSubReadReply(tid=199, attrs_read=0))", 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.003685305000001, 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46362988399999999, 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204216+0000", 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304179+0000", 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099963732 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304195+0000", 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5982999999999999e-05 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667760+0000", 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.36356425599999997 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667845+0000", 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5913000000000002e-05 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.cs0 94/84 ECSubRead(tid=200, to_read={3:3c73a4f8:::smithi09731216-45:head=0,356352,0}, subchunks={3:3c73a4f8:::smithi09731216-45:head=[0,1]}, attrs_to_read=))", 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 17.003147473999999, 2024-04-18T21:53:08.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.463402286, 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.204753+0000", 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304241+0000", 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099488015999999999 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.304269+0000", 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7447e-05 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.667948+0000", 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.36367946200000001 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:51.668156+0000", 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020736100000000001 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.3.19:5 3.5s0 3:a2c4e662:::smithi09731216-50:head [assert-version v34,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:57.306094+0000", 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.901806532, 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41520620600000002, 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306094+0000", 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306094+0000", 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306097+0000", 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5560000000000002e-06 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306109+0000", 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2327000000000001e-05 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306129+0000", 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0041999999999999e-05 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306143+0000", 2024-04-18T21:53:08.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3687e-05 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306283+0000", 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013993 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.306491+0000", 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020857499999999999 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.612374+0000", 2024-04-18T21:53:08.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30588252799999999 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.612636+0000", 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026220099999999999 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:57.721301+0000", 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10866436 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:865 3.5s0 3:a2c4e662:::smithi09731216-50:head [append 0~0] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.425631+0000", 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.7822697290000002, 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41618238899999999, 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 865 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425631+0000", 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425631+0000", 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425634+0000", 2024-04-18T21:53:08.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6630000000000002e-06 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425646+0000", 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1747e-05 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425678+0000", 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1999000000000003e-05 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425695+0000", 2024-04-18T21:53:08.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7635999999999999e-05 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425804+0000", 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010906199999999999 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425810+0000", 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4369999999999998e-06 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.526362+0000", 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10055267499999999 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.526579+0000", 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021679199999999999 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.528660+0000", 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002080391 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.630780+0000", 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102120831 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.841660+0000", 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21087966999999999 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.841814+0000", 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015348599999999999 2024-04-18T21:53:08.302 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:866 3.5s0 3:a2c4e662:::smithi09731216-50:head [rollback 1c] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.425732+0000", 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.7821684380000002, 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41635736299999998, 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.303 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 866 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425732+0000", 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425732+0000", 2024-04-18T21:53:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425733+0000", 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7700000000000003e-07 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425738+0000", 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3229999999999999e-06 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425749+0000", 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1653e-05 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425758+0000", 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3860000000000007e-06 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425848+0000", 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0006000000000005e-05 2024-04-18T21:53:08.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425851+0000", 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5949999999999999e-06 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.533729+0000", 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107877722 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.533900+0000", 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017071 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.535750+0000", 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018506180000000001 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.630783+0000", 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.095032545999999996 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.842013+0000", 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21123041200000001 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.842090+0000", 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6514999999999995e-05 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.309 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:867 3.5s0 3:a2c4e662:::smithi09731216-50:head [append 0~0] snapc 1c=[1c,1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:00.425773+0000", 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.7821277389999999, 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.41649866800000002, 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 867 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.310 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425773+0000", 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425773+0000", 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425774+0000", 2024-04-18T21:53:08.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.54e-07 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425777+0000", 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3699999999999999e-06 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425789+0000", 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1833e-05 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.425797+0000", 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3059999999999998e-06 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.525877+0000", 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100079977 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.525890+0000", 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2149e-05 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.539816+0000", 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013926122000000001 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.540218+0000", 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040234500000000002 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.541647+0000", 2024-04-18T21:53:08.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014284969999999999 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.630785+0000", 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.089138334 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.842202+0000", 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.211417461 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:00.842272+0000", 2024-04-18T21:53:08.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9519999999999998e-05 2024-04-18T21:53:08.316 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.316 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.316 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.316 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.316 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.316 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:08.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-18T21:53:08.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-18T21:53:08.319 INFO:tasks.rados.rados.0.smithi097.stdout:387: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-18T21:53:08.319 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-18T21:53:08.320 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:08.314+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:08.388 INFO:tasks.rados.rados.0.smithi097.stdout:391: finishing write tid 3 to smithi09731216-17 2024-04-18T21:53:08.389 INFO:tasks.rados.rados.0.smithi097.stdout:391: oid 17 updating version 0 to 220 2024-04-18T21:53:08.389 INFO:tasks.rados.rados.0.smithi097.stdout:391: oid 17 version 220 is already newer than 219 2024-04-18T21:53:08.389 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 17 v 220 (ObjNum 147 snap 32 seq_num 147) dirty exists 2024-04-18T21:53:08.389 INFO:tasks.rados.rados.0.smithi097.stdout:391: left oid 17 (ObjNum 147 snap 32 seq_num 147) 2024-04-18T21:53:08.397 INFO:tasks.rados.rados.0.smithi097.stdout:393: finishing write tid 1 to smithi09731216-30 2024-04-18T21:53:08.397 INFO:tasks.rados.rados.0.smithi097.stdout:393: finishing write tid 2 to smithi09731216-30 2024-04-18T21:53:08.418 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:08.414+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:08.469 INFO:tasks.rados.rados.0.smithi097.stdout:387: done (8 left) 2024-04-18T21:53:08.469 INFO:tasks.rados.rados.0.smithi097.stdout:391: done (7 left) 2024-04-18T21:53:08.469 INFO:tasks.rados.rados.0.smithi097.stdout:396: done (6 left) 2024-04-18T21:53:08.470 INFO:tasks.rados.rados.0.smithi097.stdout:397: done (5 left) 2024-04-18T21:53:08.470 INFO:tasks.rados.rados.0.smithi097.stdout:398: read oid 32 snap -1 2024-04-18T21:53:08.470 INFO:tasks.rados.rados.0.smithi097.stdout:398: expect (ObjNum 122 snap 24 seq_num 122) 2024-04-18T21:53:08.470 INFO:tasks.rados.rados.0.smithi097.stdout:399: rollback oid 44 current snap is 33 2024-04-18T21:53:08.470 INFO:tasks.rados.rados.0.smithi097.stdout:rollback oid 44 to 26 2024-04-18T21:53:08.470 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 7 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:982 3.as0 3:561bcae7:::smithi09731216-2:head [copy-from ver 181 in=55b] snapc 21=[21,20,1e,1d,1c,1b] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:08.087006+0000", 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.40889567500000001, 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.408949057, 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.496 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 982 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087006+0000", 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087006+0000", 2024-04-18T21:53:08.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087008+0000", 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.787e-06 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087014+0000", 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0990000000000004e-06 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087026+0000", 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2496e-05 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187100+0000", 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10007377200000001 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187170+0000", 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0010000000000002e-05 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187285+0000", 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000114895 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:983 3.as0 3:561bcae7:::smithi09731216-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:08.187123+0000", 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.30877904699999997, 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30895791, 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.500 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for blocked object", 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 983 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187123+0000", 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187123+0000", 2024-04-18T21:53:08.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187124+0000", 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1629999999999999e-06 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187129+0000", 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0259999999999998e-06 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187139+0000", 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8560000000000002e-06 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187145+0000", 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8499999999999999e-06 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187504+0000", 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035902800000000001 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187567+0000", 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3422000000000004e-05 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-18T21:53:08.504 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 2 2024-04-18T21:53:08.505 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:08.505 INFO:tasks.rados.rados.0.smithi097.stdout:389: finishing write tid 3 to smithi09731216-4 2024-04-18T21:53:08.505 INFO:tasks.rados.rados.0.smithi097.stdout:389: oid 4 updating version 0 to 83 2024-04-18T21:53:08.505 INFO:tasks.rados.rados.0.smithi097.stdout:389: oid 4 version 83 is already newer than 82 2024-04-18T21:53:08.505 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 4 v 83 (ObjNum 145 snap 32 seq_num 145) dirty exists 2024-04-18T21:53:08.506 INFO:tasks.rados.rados.0.smithi097.stdout:389: left oid 4 (ObjNum 145 snap 32 seq_num 145) 2024-04-18T21:53:08.506 INFO:tasks.rados.rados.0.smithi097.stdout:389: done (6 left) 2024-04-18T21:53:08.506 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 6 2024-04-18T21:53:08.506 INFO:tasks.rados.rados.0.smithi097.stdout:394: finishing copy_from racing read to smithi09731216-2 2024-04-18T21:53:08.514 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0: {} 2024-04-18T21:53:08.514 INFO:teuthology.orchestra.run.smithi097.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-18T21:53:08.516 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-18T21:53:08.519 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:53:08.514+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:53:08.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-18T21:53:08.610 INFO:tasks.rados.rados.0.smithi097.stdout:394: finishing copy_from to smithi09731216-2 2024-04-18T21:53:08.610 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 2 v 181 (ObjNum 126 snap 24 seq_num 126) dirty exists 2024-04-18T21:53:08.610 INFO:tasks.rados.rados.0.smithi097.stdout:394: done (5 left) 2024-04-18T21:53:08.610 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 5 2024-04-18T21:53:08.620 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:53:08.614+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:53:08.637 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:08.637 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-04-18T21:53:08.637 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-04-18T21:53:08.637 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-04-18T21:53:08.637 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:08.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-18T21:53:08.698 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 13 v 44 (ObjNum 143 snap 31 seq_num 143) dirty exists 2024-04-18T21:53:08.698 INFO:tasks.rados.rados.0.smithi097.stdout:395: done (4 left) 2024-04-18T21:53:08.698 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 4 2024-04-18T21:53:08.699 INFO:tasks.rados.rados.0.smithi097.stdout:388: finishing write tid 3 to smithi09731216-21 2024-04-18T21:53:08.699 INFO:tasks.rados.rados.0.smithi097.stdout:388: oid 21 updating version 0 to 77 2024-04-18T21:53:08.699 INFO:tasks.rados.rados.0.smithi097.stdout:388: oid 21 version 77 is already newer than 76 2024-04-18T21:53:08.699 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 21 v 77 (ObjNum 144 snap 32 seq_num 144) dirty exists 2024-04-18T21:53:08.699 INFO:tasks.rados.rados.0.smithi097.stdout:388: left oid 21 (ObjNum 144 snap 32 seq_num 144) 2024-04-18T21:53:08.699 INFO:tasks.rados.rados.0.smithi097.stdout:388: done (3 left) 2024-04-18T21:53:08.699 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 3 2024-04-18T21:53:08.721 INFO:tasks.ceph.osd.10.smithi183.stderr:2024-04-18T21:53:08.714+0000 7ff775811640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25338) UID: 0 2024-04-18T21:53:08.728 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1: {} 2024-04-18T21:53:08.728 INFO:teuthology.orchestra.run.smithi097.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-18T21:53:08.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-18T21:53:08.746 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2: {} 2024-04-18T21:53:08.746 INFO:teuthology.orchestra.run.smithi097.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-18T21:53:08.762 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3: {} 2024-04-18T21:53:08.763 INFO:teuthology.orchestra.run.smithi097.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-18T21:53:08.778 INFO:tasks.rados.rados.0.smithi097.stdout:399: finishing rollback tid 0 to smithi09731216-44 2024-04-18T21:53:08.779 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-18T21:53:08.779 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:308 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~737280] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 70.686934958999998, 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69365002799999997, 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.780 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 308 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090472+0000", 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.090475+0000", 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.976e-06 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091768+0000", 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00129317 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091803+0000", 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4502999999999998e-05 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.091826+0000", 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3235999999999999e-05 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270501+0000", 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.178674888 2024-04-18T21:53:08.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.270680+0000", 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017917599999999999 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.273556+0000", 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028757610000000001 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281580+0000", 2024-04-18T21:53:08.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080240799999999994 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.783886+0000", 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50230573000000001 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784122+0000", 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023650799999999999 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:309 3.as0 3:5bb4af71:::smithi09731216-7:head [append 0~376832] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 70.684853966000006, 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69216249600000002, 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 309 2024-04-18T21:53:08.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092553+0000", 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.092555+0000", 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093122+0000", 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056744499999999997 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093146+0000", 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128999999999999e-05 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093165+0000", 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8300000000000001e-05 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275352+0000", 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18218714899999999 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.275632+0000", 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028059100000000001 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.277549+0000", 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001917087 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.281584+0000", 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004034513 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784502+0000", 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50291762600000001 2024-04-18T21:53:08.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784716+0000", 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021415599999999999 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:310 3.as0 3:5bb4af71:::smithi09731216-7:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 70.684212795999997, 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69187687499999995, 2024-04-18T21:53:08.791 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 310 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093194+0000", 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093196+0000", 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6530000000000001e-06 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093210+0000", 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3517e-05 2024-04-18T21:53:08.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093227+0000", 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7050000000000001e-05 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093243+0000", 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5965999999999999e-05 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279008+0000", 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.185765398 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.279287+0000", 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027941400000000002 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.280700+0000", 2024-04-18T21:53:08.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412581 2024-04-18T21:53:08.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.482172+0000", 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20147153900000001 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.784956+0000", 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30278430299999998 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785071+0000", 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115454 2024-04-18T21:53:08.798 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:311 3.as0 3:5bb4af71:::smithi09731216-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 70.684137844000006, 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99553992999999996, 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.799 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 311 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093269+0000", 2024-04-18T21:53:08.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093271+0000", 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093278+0000", 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1459999999999999e-06 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093295+0000", 2024-04-18T21:53:08.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7408999999999999e-05 2024-04-18T21:53:08.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.093328+0000", 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2276000000000001e-05 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.785901+0000", 2024-04-18T21:53:08.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.692573628 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:58.786181+0000", 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027934999999999999 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:51:59.088809+0000", 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30262875700000003 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.805 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:429 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~0] snapc e=[e,d,c,a,9,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 54.331731409, 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51405278700000001, 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.806 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 429 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445676+0000", 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445677+0000", 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5260000000000001e-06 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445685+0000", 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2370000000000003e-06 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445703+0000", 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8295999999999999e-05 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.445718+0000", 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5383000000000002e-05 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545773+0000", 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100054752 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.545785+0000", 2024-04-18T21:53:08.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1783000000000001e-05 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557166+0000", 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011381578 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.557443+0000", 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027702800000000002 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.558675+0000", 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012320479999999999 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.659884+0000", 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101208162 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959511+0000", 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.29962765699999999 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:14.959729+0000", 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021733700000000001 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:495 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~737280] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.270367976000003, 2024-04-18T21:53:08.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.664354899, 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 495 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:53:08.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507039+0000", 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.507043+0000", 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.596e-06 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508389+0000", 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013461460000000001 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508432+0000", 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3294999999999998e-05 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.508456+0000", 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4134e-05 2024-04-18T21:53:08.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581521+0000", 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.073064403 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.581827+0000", 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030644999999999999 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.586475+0000", 2024-04-18T21:53:08.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046472989999999997 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.699953+0000", 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.113478523 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171243+0000", 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47128973499999999 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.171394+0000", 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000151318 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:496 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~573440] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.267180322999998, 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76183363900000001, 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.818 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 496 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510227+0000", 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.510229+0000", 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.007e-06 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511015+0000", 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000785792 2024-04-18T21:53:08.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511046+0000", 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0873000000000001e-05 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511061+0000", 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5293000000000001e-05 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.691993+0000", 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.180931701 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.692191+0000", 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019870099999999999 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.694402+0000", 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022106690000000002 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.701975+0000", 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075735730000000001 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.271924+0000", 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56994823500000003 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272061+0000", 2024-04-18T21:53:08.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013679499999999999 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:497 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~196608] snapc 10=[10,f,a,6] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.266130961000002, 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.76114050600000005, 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.824 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 497 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511276+0000", 2024-04-18T21:53:08.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511278+0000", 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.387e-06 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511608+0000", 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033003500000000001 2024-04-18T21:53:08.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511637+0000", 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9706e-05 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.511661+0000", 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3439999999999999e-05 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696155+0000", 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18449448800000001 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.696457+0000", 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000301496 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.698580+0000", 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021231879999999998 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:20.702633+0000", 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040534079999999997 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272305+0000", 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56967170199999995 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:21.272417+0000", 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111656 2024-04-18T21:53:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:535 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~802816] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.378922248000002, 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60500271100000003, 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.830 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 535 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398485+0000", 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.398487+0000", 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0650000000000001e-06 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399399+0000", 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091160899999999997 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399417+0000", 2024-04-18T21:53:08.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7915999999999999e-05 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.399426+0000", 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7070000000000004e-06 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.485946+0000", 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086519677000000003 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.486140+0000", 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019389 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.488798+0000", 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002657707 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.600548+0000", 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11175003 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003305+0000", 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40275720799999998 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003488+0000", 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000182902 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:536 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~638976] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:53:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.373944590999997, 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60040761799999998, 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 536 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.403463+0000", 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.09e-07 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404080+0000", 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00061689199999999996 2024-04-18T21:53:08.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404092+0000", 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2055000000000001e-05 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404101+0000", 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5650000000000002e-06 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491527+0000", 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087426312000000006 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.491697+0000", 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017024799999999999 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.493862+0000", 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021641299999999998 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700761+0000", 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20689907199999999 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003718+0000", 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30295725099999998 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.003870+0000", 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015228400000000001 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.840 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:537 3.10s0 3:0e754656:::smithi09731216-19:head [append 0~131072] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.372974929999998, 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.59990886700000001, 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.841 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 537 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404432+0000", 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404433+0000", 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.44e-07 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404610+0000", 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017705100000000001 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404620+0000", 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8050000000000001e-06 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404625+0000", 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5029999999999997e-06 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495333+0000", 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.090708141000000006 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.495532+0000", 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001983 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.496836+0000", 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013041789999999999 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700772+0000", 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20393593900000001 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004176+0000", 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30340381399999999 2024-04-18T21:53:08.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.004341+0000", 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016549100000000001 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:538 3.10s0 3:0e754656:::smithi09731216-19:head [setxattr _header (57) in=7b] snapc 13=[12,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.372772943000001, 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70001739100000004, 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.846 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 538 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404634+0000", 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404635+0000", 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6699999999999999e-07 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404637+0000", 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.683e-06 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404642+0000", 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.967e-06 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404647+0000", 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.25e-06 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.497929+0000", 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.093281966999999993 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498095+0000", 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016633800000000001 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.498905+0000", 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081039199999999995 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.700775+0000", 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20186959700000001 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104337+0000", 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40356159400000002 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.104652+0000", 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031513599999999999 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:539 3.10s0 3:0e754656:::smithi09731216-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-04-18T21:53:08.851 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.372754868999998, 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90384237700000003, 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 539 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404652+0000", 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404653+0000", 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0600000000000001e-07 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404655+0000", 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.95e-06 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404662+0000", 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9530000000000002e-06 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.404666+0000", 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4089999999999996e-06 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499383+0000", 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094717379000000004 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:25.499540+0000", 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015672099999999999 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.105824+0000", 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60628387699999997 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.106088+0000", 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026422700000000002 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:26.308495+0000", 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20240645500000001 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:572 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~507904 in=507904b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:08.856 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 38.244185733999998, 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.60481051399999997, 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 572 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533221+0000", 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.533224+0000", 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.824e-06 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534169+0000", 2024-04-18T21:53:08.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094509200000000002 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534205+0000", 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5482999999999999e-05 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534228+0000", 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2806000000000002e-05 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634530+0000", 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10030222799999999 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.634798+0000", 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026831400000000002 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.638371+0000", 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035731249999999999 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744705+0000", 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106333831 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.137882+0000", 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39317701599999999 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.138032+0000", 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014979500000000001 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.861 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:573 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~606208 in=606208b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 38.243144426000001, 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70430669099999998, 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.862 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 573 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534263+0000", 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.534265+0000", 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9800000000000001e-06 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535350+0000", 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010856869999999999 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535392+0000", 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1192999999999997e-05 2024-04-18T21:53:08.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535427+0000", 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5559999999999998e-05 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642525+0000", 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107097461 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.642617+0000", 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2181000000000004e-05 2024-04-18T21:53:08.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.643719+0000", 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011017449999999999 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747184+0000", 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103465025 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238444+0000", 2024-04-18T21:53:08.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49126059700000002 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238569+0000", 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012526200000000001 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:574 3.as0 3:561bcae7:::smithi09731216-2:head [append 0~557056 in=557056b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 38.241949486000003, 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.70342813999999998, 2024-04-18T21:53:08.867 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 574 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535458+0000", 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.535460+0000", 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.796e-06 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540737+0000", 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052778449999999998 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540774+0000", 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6372999999999999e-05 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540799+0000", 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5743999999999999e-05 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744797+0000", 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20399719899999999 2024-04-18T21:53:08.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.744963+0000", 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016631799999999999 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.746246+0000", 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012830319999999999 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.747841+0000", 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001595352 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238775+0000", 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49093394899999998 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.238886+0000", 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110532 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.872 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:575 3.as0 3:561bcae7:::smithi09731216-2:head [setxattr _header (56) in=63b] snapc 14=[14,11,10,a,6] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 38.236569033000002, 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.69838045999999998, 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 575 2024-04-18T21:53:08.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540838+0000", 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540841+0000", 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4540000000000001e-06 2024-04-18T21:53:08.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540850+0000", 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2329999999999995e-06 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540874+0000", 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3980000000000001e-05 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.540893+0000", 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9173000000000001e-05 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748051+0000", 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20715831100000001 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748150+0000", 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9155999999999997e-05 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.748574+0000", 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042389100000000002 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:30.753028+0000", 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044531889999999998 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.239064+0000", 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.48603637100000002 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:31.239219+0000", 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154702 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 82/78 [PushReplyOp(3:0e754656:::smithi09731216-19:10)])", 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 29.482564763999999, 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81294819200000001, 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.878 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294842+0000", 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294850+0000", 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6299999999999998e-06 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294862+0000", 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.186e-05 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294892+0000", 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0103e-05 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.294914+0000", 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1633000000000001e-05 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.107740+0000", 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.81282626999999996 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.107791+0000", 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0695999999999997e-05 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(3.10s0 82/78 [PushReplyOp(3:0e754656:::smithi09731216-19:10)])", 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 29.48168562, 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91227883099999996, 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:53:08.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295722+0000", 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295723+0000", 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.703e-06 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295734+0000", 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.064e-05 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295762+0000", 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7606999999999998e-05 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:39.295776+0000", 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.402e-05 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.207972+0000", 2024-04-18T21:53:08.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91219607400000002 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:52:40.208000+0000", 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8787000000000001e-05 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4509.0:982 3.as0 3:561bcae7:::smithi09731216-2:head [copy-from ver 181 in=55b] snapc 21=[21,20,1e,1d,1c,1b] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-04-18T21:53:08.087006+0000", 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.69040114900000005, 2024-04-18T21:53:08.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52386571999999998, 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4509", 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.97:0/2929837246", 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 982 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087006+0000", 2024-04-18T21:53:08.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087006+0000", 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087008+0000", 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.787e-06 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-04-18T21:53:08.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087014+0000", 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0990000000000004e-06 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.087026+0000", 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2496e-05 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187100+0000", 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10007377200000001 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-04-18T21:53:08.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187170+0000", 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0010000000000002e-05 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.187285+0000", 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000114895 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.501723+0000", 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31443833100000002 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2024-04-18T21:53:08.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.505250+0000", 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035269099999999999 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.610733+0000", 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10548310900000001 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-04-18T21:53:08.610872+0000", 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013831100000000001 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.890 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.891 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-18T21:53:08.891 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-18T21:53:08.891 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-18T21:53:08.892 INFO:tasks.rados.rados.0.smithi097.stdout:399: finishing rollback tid 1 to smithi09731216-44 2024-04-18T21:53:08.892 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4: {} 2024-04-18T21:53:08.892 INFO:teuthology.orchestra.run.smithi097.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-18T21:53:08.893 INFO:tasks.rados.rados.0.smithi097.stdout:393: finishing write tid 3 to smithi09731216-30 2024-04-18T21:53:08.893 INFO:tasks.rados.rados.0.smithi097.stdout:393: oid 30 updating version 0 to 42 2024-04-18T21:53:08.893 INFO:tasks.rados.rados.0.smithi097.stdout:393: oid 30 updating version 42 to 43 2024-04-18T21:53:08.893 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 30 v 43 (ObjNum 149 snap 32 seq_num 149) dirty exists 2024-04-18T21:53:08.893 INFO:tasks.rados.rados.0.smithi097.stdout:393: left oid 30 (ObjNum 149 snap 32 seq_num 149) 2024-04-18T21:53:08.893 INFO:tasks.rados.rados.0.smithi097.stdout:393: done (2 left) 2024-04-18T21:53:08.893 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 2 2024-04-18T21:53:08.894 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5: {} 2024-04-18T21:53:08.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-18T21:53:08.894 INFO:teuthology.orchestra.run.smithi097.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-18T21:53:08.895 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6: {} 2024-04-18T21:53:08.895 INFO:teuthology.orchestra.run.smithi097.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-18T21:53:08.895 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7: {} 2024-04-18T21:53:08.896 INFO:teuthology.orchestra.run.smithi097.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-18T21:53:08.896 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8: {} 2024-04-18T21:53:08.896 INFO:teuthology.orchestra.run.smithi097.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-18T21:53:08.897 INFO:tasks.rados.rados.0.smithi097.stdout:399: finishing rollback tid 2 to smithi09731216-44 2024-04-18T21:53:08.897 INFO:tasks.rados.rados.0.smithi097.stdout:update_object_version oid 44 v 128 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-18T21:53:08.897 INFO:tasks.rados.rados.0.smithi097.stdout:399: done (1 left) 2024-04-18T21:53:08.897 INFO:tasks.rados.rados.0.smithi097.stdout: waiting on 1 2024-04-18T21:53:08.898 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:08.886+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:08.900 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9: {} 2024-04-18T21:53:08.900 INFO:teuthology.orchestra.run.smithi097.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-18T21:53:08.903 INFO:tasks.rados.rados.0.smithi097.stdout:398: expect (ObjNum 122 snap 24 seq_num 122) 2024-04-18T21:53:08.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-04-18T21:53:08.922 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10: {} 2024-04-18T21:53:08.923 INFO:teuthology.orchestra.run.smithi097.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-18T21:53:08.944 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11: {} 2024-04-18T21:53:08.944 INFO:teuthology.orchestra.run.smithi097.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-18T21:53:08.979 INFO:tasks.rados.rados.0.smithi097.stdout:398: done (0 left) 2024-04-18T21:53:08.984 INFO:tasks.rados.rados.0.smithi097.stderr:0 errors. 2024-04-18T21:53:08.984 INFO:tasks.rados.rados.0.smithi097.stderr: 2024-04-18T21:53:08.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T21:53:08.992 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:53:08.986+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:53:09.026 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:09.027 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:09.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-18T21:53:09.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 reset_purged_snaps_last 2024-04-18T21:53:09.094 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:09.090+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:09.194 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:09.190+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:09.205 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:09.205 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:09.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-18T21:53:09.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 reset_purged_snaps_last 2024-04-18T21:53:09.325 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:09.325 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":112,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:53:08.460874+0000","last_up_change":"2024-04-18T21:51:29.769135+0000","last_in_change":"2024-04-18T21:52:45.415971+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T21:51:28.321879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-18T21:51:30.835497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-18T21:51:40.934866+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":8,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":20,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"3.16","ready_epoch":110,"last_epoch_started":109,"last_epoch_clean":109,"source_version":"105'19","target_version":"100'23"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"112","last_force_op_resend_preluminous":"112","auid":0,"snap_mode":"selfmanaged","snap_seq":34,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lrcprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6812","nonce":3458443931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6817","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6820","nonce":3458443931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6831","nonce":3458443931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6828","nonce":3458443931}]},"public_addr":"172.21.15.97:6812/3458443931","cluster_addr":"172.21.15.97:6820/3458443931","heartbeat_back_addr":"172.21.15.97:6831/3458443931","heartbeat_front_addr":"172.21.15.97:6828/3458443931","state":["exists","up"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6801","nonce":1179456160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6803","nonce":1179456160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6811","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6815","nonce":1179456160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6807","nonce":1179456160}]},"public_addr":"172.21.15.97:6801/1179456160","cluster_addr":"172.21.15.97:6803/1179456160","heartbeat_back_addr":"172.21.15.97:6815/1179456160","heartbeat_front_addr":"172.21.15.97:6807/1179456160","state":["exists","up"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6813","nonce":598211345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6819","nonce":598211345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6827","nonce":598211345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6821","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6823","nonce":598211345}]},"public_addr":"172.21.15.97:6813/598211345","cluster_addr":"172.21.15.97:6819/598211345","heartbeat_back_addr":"172.21.15.97:6827/598211345","heartbeat_front_addr":"172.21.15.97:6823/598211345","state":["exists","up"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6806","nonce":40393780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6814","nonce":40393780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6829","nonce":40393780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6822","nonce":40393780}]},"public_addr":"172.21.15.97:6806/40393780","cluster_addr":"172.21.15.97:6814/40393780","heartbeat_back_addr":"172.21.15.97:6829/40393780","heartbeat_front_addr":"172.21.15.97:6822/40393780","state":["exists","up"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6809","nonce":3973913623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6811","nonce":3973913623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6815","nonce":3973913623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6813","nonce":3973913623}]},"public_addr":"172.21.15.104:6809/3973913623","cluster_addr":"172.21.15.104:6811/3973913623","heartbeat_back_addr":"172.21.15.104:6815/3973913623","heartbeat_front_addr":"172.21.15.104:6813/3973913623","state":["exists","up"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6801","nonce":1478992259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6803","nonce":1478992259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6807","nonce":1478992259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6805","nonce":1478992259}]},"public_addr":"172.21.15.104:6801/1478992259","cluster_addr":"172.21.15.104:6803/1478992259","heartbeat_back_addr":"172.21.15.104:6807/1478992259","heartbeat_front_addr":"172.21.15.104:6805/1478992259","state":["exists","up"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6817","nonce":462814955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6819","nonce":462814955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6823","nonce":462814955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6821","nonce":462814955}]},"public_addr":"172.21.15.104:6817/462814955","cluster_addr":"172.21.15.104:6819/462814955","heartbeat_back_addr":"172.21.15.104:6823/462814955","heartbeat_front_addr":"172.21.15.104:6821/462814955","state":["exists","up"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6825","nonce":581913590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6827","nonce":581913590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6831","nonce":581913590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6829","nonce":581913590}]},"public_addr":"172.21.15.104:6825/581913590","cluster_addr":"172.21.15.104:6827/581913590","heartbeat_back_addr":"172.21.15.104:6831/581913590","heartbeat_front_addr":"172.21.15.104:6829/581913590","state":["exists","up"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6806","nonce":1020504720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6812","nonce":1020504720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6825","nonce":1020504720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6819","nonce":1020504720}]},"public_addr":"172.21.15.183:6806/1020504720","cluster_addr":"172.21.15.183:6812/1020504720","heartbeat_back_addr":"172.21.15.183:6825/1020504720","heartbeat_front_addr":"172.21.15.183:6819/1020504720","state":["exists","up"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6814","nonce":1194329140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6821","nonce":1194329140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6829","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6830","nonce":1194329140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6827","nonce":1194329140}]},"public_addr":"172.21.15.183:6814/1194329140","cluster_addr":"172.21.15.183:6821/1194329140","heartbeat_back_addr":"172.21.15.183:6830/1194329140","heartbeat_front_addr":"172.21.15.183:6827/1194329140","state":["exists","up"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6813","nonce":1192340806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6817","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6820","nonce":1192340806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6831","nonce":1192340806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6823","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6826","nonce":1192340806}]},"public_addr":"172.21.15.183:6813/1192340806","cluster_addr":"172.21.15.183:6820/1192340806","heartbeat_back_addr":"172.21.15.183:6831/1192340806","heartbeat_front_addr":"172.21.15.183:6826/1192340806","state":["exists","up"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6801","nonce":3144695911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6803","nonce":3144695911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6815","nonce":3144695911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6807","nonce":3144695911}]},"public_addr":"172.21.15.183:6801/3144695911","cluster_addr":"172.21.15.183:6803/3144695911","heartbeat_back_addr":"172.21.15.183:6815/3144695911","heartbeat_front_addr":"172.21.15.183:6807/3144695911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-18T21:52:32.843393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:55.891649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:56.078049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:45.523740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:35.646618+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:43.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:36.263252+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-18T21:52:28.081472+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:42.792903+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:38.930203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:35.599364+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:47.013093+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.2","osds":[2,6,0,9,11,8,1,4]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.9","osds":[3,2,7,1,5,9,4,6]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":16,"length":1},{"begin":21,"length":6},{"begin":31,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:53:09.326 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-04-18T21:53:09.322+0000 7f8753bc6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25108) UID: 0 2024-04-18T21:53:09.337 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:09.337 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:09.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:53:09.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-18T21:53:09.397 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:09.394+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:09.484 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:09.485 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:09.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 reset_purged_snaps_last 2024-04-18T21:53:09.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:53:09.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-18T21:53:09.533 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-04-18T21:53:09.530+0000 7f2bfd1b8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25110) UID: 0 2024-04-18T21:53:09.597 INFO:tasks.ceph.osd.9.smithi183.stderr:2024-04-18T21:53:09.594+0000 7f9d9195b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25336) UID: 0 2024-04-18T21:53:09.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 reset_purged_snaps_last 2024-04-18T21:53:09.665 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:09.665 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:09.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-18T21:53:09.693 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:09.694 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:53:09.698 INFO:tasks.ceph.osd.7.smithi104.stderr:2024-04-18T21:53:09.694+0000 7efd2cd00640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25099) UID: 0 2024-04-18T21:53:09.710 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-04-18T21:53:09.475001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104596000,"num_objects":60,"num_object_clones":18,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":1,"num_read":439,"num_read_kb":164906,"num_write":743,"num_write_kb":127530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":158241312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":207,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":943718400,"kb_used":555404,"kb_used_data":210304,"kb_used_omap":15,"kb_used_meta":344944,"kb_avail":943162996,"statfs":{"total":966367641600,"available":965798907904,"internally_reserved":0,"allocated":215351296,"data_stored":292698287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15954,"internal_metadata":353223086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,8,4,1,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15171584,"num_objects":8,"num_object_clones":7,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":13,"num_read_kb":6964,"num_write":23,"num_write_kb":14816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.100197"},"pg_stats":[{"pgid":"3.15","version":"98'15","reported_seq":429,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.476567+0000","last_change":"2024-04-18T21:53:01.442934+0000","last_active":"2024-04-18T21:53:04.476567+0000","last_peered":"2024-04-18T21:53:04.476567+0000","last_clean":"2024-04-18T21:53:04.476567+0000","last_became_active":"2024-04-18T21:52:52.582785+0000","last_became_peered":"2024-04-18T21:52:52.582785+0000","last_unstale":"2024-04-18T21:53:04.476567+0000","last_undegraded":"2024-04-18T21:53:04.476567+0000","last_fullsized":"2024-04-18T21:53:04.476567+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'4","last_scrub_stamp":"2024-04-18T21:51:51.020140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:51.020140+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0.001004808,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1277952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,4,11,3,9],"acting":[5,8,10,6,4,11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.14","version":"98'17","reported_seq":286,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-18T21:53:04.791419+0000","last_change":"2024-04-18T21:53:04.791419+0000","last_active":"2024-04-18T21:53:04.791419+0000","last_peered":"2024-04-18T21:53:04.791419+0000","last_clean":"2024-04-18T21:52:54.738834+0000","last_became_active":"2024-04-18T21:53:04.690969+0000","last_became_peered":"2024-04-18T21:53:04.690969+0000","last_unstale":"2024-04-18T21:53:04.791419+0000","last_undegraded":"2024-04-18T21:53:04.382988+0000","last_fullsized":"2024-04-18T21:53:04.381604+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":5,"last_scrub":"32'4","last_scrub_stamp":"2024-04-18T21:51:52.378546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:52.378546+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1515,"objects_trimmed":2,"snaptrim_duration":0.369715556,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,6,9,11],"acting":[2147483647,2,1,5,8,6,9,2147483647],"avail_no_missing":["2(1)","1(7)","4(0)","5(4)","6(5)","9(6)","10(2)","11(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":2,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"2.7","version":"0'0","reported_seq":198,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-18T21:53:06.567622+0000","last_change":"2024-04-18T21:53:01.231628+0000","last_active":"2024-04-18T21:53:06.567622+0000","last_peered":"2024-04-18T21:53:06.567622+0000","last_clean":"2024-04-18T21:53:06.567622+0000","last_became_active":"2024-04-18T21:52:28.498662+0000","last_became_peered":"2024-04-18T21:52:28.498662+0000","last_unstale":"2024-04-18T21:53:06.567622+0000","last_undegraded":"2024-04-18T21:53:06.567622+0000","last_fullsized":"2024-04-18T21:53:06.567622+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:01.231382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:01.231382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:25.311906+0000","scrub_duration":567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"105'19","reported_seq":436,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-18T21:53:08.478141+0000","last_change":"2024-04-18T21:53:08.179362+0000","last_active":"2024-04-18T21:53:08.478141+0000","last_peered":"2024-04-18T21:53:08.478141+0000","last_clean":"2024-04-18T21:53:08.478141+0000","last_became_active":"2024-04-18T21:53:07.973602+0000","last_became_peered":"2024-04-18T21:53:07.973602+0000","last_unstale":"2024-04-18T21:53:08.478141+0000","last_undegraded":"2024-04-18T21:53:08.478141+0000","last_fullsized":"2024-04-18T21:53:08.478141+0000","mapping_epoch":98,"log_start":"105'19","ondisk_log_start":"105'19","created":26,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"32'2","last_scrub_stamp":"2024-04-18T21:52:02.163308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.163308+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":47,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:31.798095+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0.10257066500000001,"stat_sum":{"num_bytes":32768,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":12579,"num_write":30,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4,6,10,2,1,5],"acting":[11,3,4,6,10,2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"},{"start":"1f","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":186,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.485334+0000","last_change":"2024-04-18T21:52:28.588502+0000","last_active":"2024-04-18T21:53:04.485334+0000","last_peered":"2024-04-18T21:53:04.485334+0000","last_clean":"2024-04-18T21:53:04.485334+0000","last_became_active":"2024-04-18T21:52:28.587613+0000","last_became_peered":"2024-04-18T21:52:28.587613+0000","last_unstale":"2024-04-18T21:53:04.485334+0000","last_undegraded":"2024-04-18T21:53:04.485334+0000","last_fullsized":"2024-04-18T21:53:04.485334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'13","reported_seq":407,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-18T21:53:08.589816+0000","last_change":"2024-04-18T21:53:07.581088+0000","last_active":"2024-04-18T21:53:08.589816+0000","last_peered":"2024-04-18T21:53:08.589816+0000","last_clean":"2024-04-18T21:53:08.589816+0000","last_became_active":"2024-04-18T21:53:03.679247+0000","last_became_peered":"2024-04-18T21:53:03.679247+0000","last_unstale":"2024-04-18T21:53:08.589816+0000","last_undegraded":"2024-04-18T21:53:08.589816+0000","last_fullsized":"2024-04-18T21:53:08.589816+0000","mapping_epoch":89,"log_start":"100'13","ondisk_log_start":"100'13","created":26,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:49.709009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:49.709009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":13,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1159,"objects_trimmed":0,"snaptrim_duration":0.0012441889999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,10,9,1,4,5],"acting":[8,3,2,10,9,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"},{"start":"1f","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.391139+0000","last_change":"2024-04-18T21:53:03.628642+0000","last_active":"2024-04-18T21:53:04.391139+0000","last_peered":"2024-04-18T21:53:04.391139+0000","last_clean":"2024-04-18T21:53:04.391139+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:53:04.391139+0000","last_undegraded":"2024-04-18T21:53:04.391139+0000","last_fullsized":"2024-04-18T21:53:04.391139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:03.628514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.628514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:24.839453+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"100'11","reported_seq":468,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-18T21:53:05.488120+0000","last_change":"2024-04-18T21:53:01.548714+0000","last_active":"2024-04-18T21:53:05.488120+0000","last_peered":"2024-04-18T21:53:05.488120+0000","last_clean":"2024-04-18T21:53:05.488120+0000","last_became_active":"2024-04-18T21:52:53.588517+0000","last_became_peered":"2024-04-18T21:52:53.588517+0000","last_unstale":"2024-04-18T21:53:05.488120+0000","last_undegraded":"2024-04-18T21:53:05.488120+0000","last_fullsized":"2024-04-18T21:53:05.488120+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'11","last_scrub_stamp":"2024-04-18T21:53:01.044881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:53:01.044881+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:01.885052+0000","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0.002784173,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":3170,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3244032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,1,5,8,6,9,11],"acting":[10,2,1,5,8,6,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-18T21:53:08.677804+0000","last_change":"2024-04-18T21:53:00.433742+0000","last_active":"2024-04-18T21:53:08.677804+0000","last_peered":"2024-04-18T21:53:08.677804+0000","last_clean":"2024-04-18T21:53:08.677804+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:53:08.677804+0000","last_undegraded":"2024-04-18T21:53:08.677804+0000","last_fullsized":"2024-04-18T21:53:08.677804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.433398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.433398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.629092+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"105'48","reported_seq":482,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.376858+0000","last_change":"2024-04-18T21:53:03.389888+0000","last_active":"2024-04-18T21:53:04.376858+0000","last_peered":"2024-04-18T21:53:04.376858+0000","last_clean":"2024-04-18T21:53:04.376858+0000","last_became_active":"2024-04-18T21:52:55.783534+0000","last_became_peered":"2024-04-18T21:52:55.783534+0000","last_unstale":"2024-04-18T21:53:04.376858+0000","last_undegraded":"2024-04-18T21:53:04.376858+0000","last_fullsized":"2024-04-18T21:53:04.376858+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'48","last_scrub_stamp":"2024-04-18T21:53:03.389472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.389472+0000","objects_scrubbed":2,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:19.427155+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0.40762110800000001,"stat_sum":{"num_bytes":3768320,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":11092,"num_write":32,"num_write_kb":3680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8323072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,10,6,4,11,3,9],"acting":[5,8,10,6,4,11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.3","version":"110'216","reported_seq":748,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-18T21:53:06.781397+0000","last_change":"2024-04-18T21:53:01.642735+0000","last_active":"2024-04-18T21:53:06.781397+0000","last_peered":"2024-04-18T21:53:06.781397+0000","last_clean":"2024-04-18T21:53:06.781397+0000","last_became_active":"2024-04-18T21:52:50.800315+0000","last_became_peered":"2024-04-18T21:52:50.800315+0000","last_unstale":"2024-04-18T21:53:06.781397+0000","last_undegraded":"2024-04-18T21:53:06.781397+0000","last_fullsized":"2024-04-18T21:53:06.781397+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'58","last_scrub_stamp":"2024-04-18T21:52:03.915160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.915160+0000","objects_scrubbed":16,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:22.178052+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.207546761,"stat_sum":{"num_bytes":30687232,"num_objects":11,"num_object_clones":4,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":65,"num_read_kb":57862,"num_write":62,"num_write_kb":32160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":25001984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11,6,1,5,2,3],"acting":[4,9,11,6,1,5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"2.2","version":"0'0","reported_seq":190,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.376076+0000","last_change":"2024-04-18T21:52:59.204140+0000","last_active":"2024-04-18T21:53:04.376076+0000","last_peered":"2024-04-18T21:53:04.376076+0000","last_clean":"2024-04-18T21:53:04.376076+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:53:04.376076+0000","last_undegraded":"2024-04-18T21:53:04.376076+0000","last_fullsized":"2024-04-18T21:53:04.376076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:52:59.203898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:52:59.203898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:15.459734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"109'68","reported_seq":497,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-18T21:53:05.801965+0000","last_change":"2024-04-18T21:53:01.546099+0000","last_active":"2024-04-18T21:53:05.801965+0000","last_peered":"2024-04-18T21:53:05.801965+0000","last_clean":"2024-04-18T21:53:05.801965+0000","last_became_active":"2024-04-18T21:52:45.939975+0000","last_became_peered":"2024-04-18T21:52:45.939975+0000","last_unstale":"2024-04-18T21:53:05.801965+0000","last_undegraded":"2024-04-18T21:53:05.801965+0000","last_fullsized":"2024-04-18T21:53:05.801965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'14","last_scrub_stamp":"2024-04-18T21:52:01.703221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:01.703221+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:25.916161+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.0051654609999999997,"stat_sum":{"num_bytes":8634368,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":5142,"num_write":18,"num_write_kb":9392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5373952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,11,3,8,9,5],"acting":[1,2,6,11,3,8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"2.1","version":"0'0","reported_seq":199,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.414918+0000","last_change":"2024-04-18T21:53:04.014699+0000","last_active":"2024-04-18T21:53:04.414918+0000","last_peered":"2024-04-18T21:53:04.414918+0000","last_clean":"2024-04-18T21:53:04.414918+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:53:04.414918+0000","last_undegraded":"2024-04-18T21:53:04.414918+0000","last_fullsized":"2024-04-18T21:53:04.414918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:04.014520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:04.014520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:23.188349+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"112'44","reported_seq":410,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-18T21:53:08.781967+0000","last_change":"2024-04-18T21:53:07.471301+0000","last_active":"2024-04-18T21:53:08.781967+0000","last_peered":"2024-04-18T21:53:08.781967+0000","last_clean":"2024-04-18T21:53:08.781967+0000","last_became_active":"2024-04-18T21:52:44.644662+0000","last_became_peered":"2024-04-18T21:52:44.644662+0000","last_unstale":"2024-04-18T21:53:08.781967+0000","last_undegraded":"2024-04-18T21:53:08.781967+0000","last_fullsized":"2024-04-18T21:53:08.781967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'40","last_scrub_stamp":"2024-04-18T21:53:02.348321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:53:02.348321+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:07.116262+0000","scrub_duration":1187,"objects_trimmed":0,"snaptrim_duration":0.003937571,"stat_sum":{"num_bytes":8863744,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":531,"num_write":56,"num_write_kb":9184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,3,9,5,2,6,8],"acting":[11,4,3,9,5,2,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"},{"start":"1f","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-18T21:53:05.409307+0000","last_change":"2024-04-18T21:52:46.543104+0000","last_active":"2024-04-18T21:53:05.409307+0000","last_peered":"2024-04-18T21:53:05.409307+0000","last_clean":"2024-04-18T21:53:05.409307+0000","last_became_active":"2024-04-18T21:52:46.542762+0000","last_became_peered":"2024-04-18T21:52:46.542762+0000","last_unstale":"2024-04-18T21:53:05.409307+0000","last_undegraded":"2024-04-18T21:53:05.409307+0000","last_fullsized":"2024-04-18T21:53:05.409307+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:51:31.787097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"109'40","reported_seq":420,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-18T21:53:06.262253+0000","last_change":"2024-04-18T21:53:01.554530+0000","last_active":"2024-04-18T21:53:06.262253+0000","last_peered":"2024-04-18T21:53:06.262253+0000","last_clean":"2024-04-18T21:53:06.262253+0000","last_became_active":"2024-04-18T21:52:41.694736+0000","last_became_peered":"2024-04-18T21:52:41.694736+0000","last_unstale":"2024-04-18T21:53:06.262253+0000","last_undegraded":"2024-04-18T21:53:06.262253+0000","last_fullsized":"2024-04-18T21:53:06.262253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-18T21:52:02.588772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588772+0000","objects_scrubbed":1,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:08.083591+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0.0066657089999999997,"stat_sum":{"num_bytes":4423680,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":17,"num_write_kb":5680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6,4,5,8,1,11],"acting":[10,2,6,4,5,8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"2.3","version":"0'0","reported_seq":200,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.576498+0000","last_change":"2024-04-18T21:53:00.139630+0000","last_active":"2024-04-18T21:53:04.576498+0000","last_peered":"2024-04-18T21:53:04.576498+0000","last_clean":"2024-04-18T21:53:04.576498+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:53:04.576498+0000","last_undegraded":"2024-04-18T21:53:04.576498+0000","last_fullsized":"2024-04-18T21:53:04.576498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:03.266144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":354,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-18T21:53:06.565677+0000","last_change":"2024-04-18T21:53:03.383085+0000","last_active":"2024-04-18T21:53:06.565677+0000","last_peered":"2024-04-18T21:53:06.565677+0000","last_clean":"2024-04-18T21:53:06.565677+0000","last_became_active":"2024-04-18T21:52:49.685989+0000","last_became_peered":"2024-04-18T21:52:49.685989+0000","last_unstale":"2024-04-18T21:53:06.565677+0000","last_undegraded":"2024-04-18T21:53:06.565677+0000","last_fullsized":"2024-04-18T21:53:06.565677+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'137","last_scrub_stamp":"2024-04-18T21:53:03.382665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.382665+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:13.609239+0000","scrub_duration":703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"106'55","reported_seq":428,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.390258+0000","last_change":"2024-04-18T21:53:01.749381+0000","last_active":"2024-04-18T21:53:04.390258+0000","last_peered":"2024-04-18T21:53:04.390258+0000","last_clean":"2024-04-18T21:53:04.390258+0000","last_became_active":"2024-04-18T21:52:43.857309+0000","last_became_peered":"2024-04-18T21:52:43.857309+0000","last_unstale":"2024-04-18T21:53:04.390258+0000","last_undegraded":"2024-04-18T21:53:04.390258+0000","last_fullsized":"2024-04-18T21:53:04.390258+0000","mapping_epoch":74,"log_start":"81'26","ondisk_log_start":"81'26","created":26,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-04-18T21:52:00.378956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:00.378956+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":28,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:17.059495+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0.21426530699999999,"stat_sum":{"num_bytes":9093120,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":15567,"num_write":37,"num_write_kb":6416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3506176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2,4,9,5,6,1],"acting":[3,10,2,4,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.9","version":"98'32","reported_seq":390,"reported_epoch":108,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-18T21:53:04.391806+0000","last_change":"2024-04-18T21:52:48.654275+0000","last_active":"2024-04-18T21:53:04.391806+0000","last_peered":"2024-04-18T21:53:04.391806+0000","last_clean":"2024-04-18T21:52:44.521893+0000","last_became_active":"2024-04-18T21:52:48.237783+0000","last_became_peered":"2024-04-18T21:52:48.237783+0000","last_unstale":"2024-04-18T21:53:04.391806+0000","last_undegraded":"2024-04-18T21:53:04.391806+0000","last_fullsized":"2024-04-18T21:53:04.391806+0000","mapping_epoch":90,"log_start":"64'24","ondisk_log_start":"64'24","created":26,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'6","last_scrub_stamp":"2024-04-18T21:52:02.588631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:02.588631+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":26,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0.77944305899999999,"stat_sum":{"num_bytes":1851392,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":1621,"num_write":7,"num_write_kb":1696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,1,5,9,4,10],"acting":[3,2,7,1,5,9,4,6],"avail_no_missing":["3(0)","1(3)","2(1)","4(6)","5(4)","6(7)","7(2)","9(5)"],"object_location_counts":[{"shards":"1(3),2(1),3(0),4(6),5(4),6(7),7(2),9(5)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"98'40","reported_seq":420,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-18T21:53:06.566934+0000","last_change":"2024-04-18T21:53:01.440717+0000","last_active":"2024-04-18T21:53:06.566934+0000","last_peered":"2024-04-18T21:53:06.566934+0000","last_clean":"2024-04-18T21:53:06.566934+0000","last_became_active":"2024-04-18T21:52:52.328941+0000","last_became_peered":"2024-04-18T21:52:52.328941+0000","last_unstale":"2024-04-18T21:53:06.566934+0000","last_undegraded":"2024-04-18T21:53:06.566934+0000","last_fullsized":"2024-04-18T21:53:06.566934+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:07.818999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071602600000000003,"stat_sum":{"num_bytes":884736,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":3269,"num_write":14,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3735552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,2,3,1,8,10],"acting":[6,4,9,2,3,1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.b","version":"105'66","reported_seq":462,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-18T21:53:05.587145+0000","last_change":"2024-04-18T21:53:01.647528+0000","last_active":"2024-04-18T21:53:05.587145+0000","last_peered":"2024-04-18T21:53:05.587145+0000","last_clean":"2024-04-18T21:53:05.587145+0000","last_became_active":"2024-04-18T21:52:52.802866+0000","last_became_peered":"2024-04-18T21:52:52.802866+0000","last_unstale":"2024-04-18T21:53:05.587145+0000","last_undegraded":"2024-04-18T21:53:05.587145+0000","last_fullsized":"2024-04-18T21:53:05.587145+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.19992480300000001,"stat_sum":{"num_bytes":6029312,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":13018,"num_write":26,"num_write_kb":8432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4833280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4,9,1,2,8,6],"acting":[10,3,4,9,1,2,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.c","version":"105'93","reported_seq":518,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.377083+0000","last_change":"2024-04-18T21:53:01.650832+0000","last_active":"2024-04-18T21:53:04.377083+0000","last_peered":"2024-04-18T21:53:04.377083+0000","last_clean":"2024-04-18T21:53:04.377083+0000","last_became_active":"2024-04-18T21:52:41.700367+0000","last_became_peered":"2024-04-18T21:52:41.700367+0000","last_unstale":"2024-04-18T21:53:04.377083+0000","last_undegraded":"2024-04-18T21:53:04.377083+0000","last_fullsized":"2024-04-18T21:53:04.377083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20857585100000001,"stat_sum":{"num_bytes":2097152,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":41,"num_read_kb":20973,"num_write":70,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":9289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,1,11,10,9,2],"acting":[5,8,6,1,11,10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.d","version":"0'0","reported_seq":124,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-18T21:53:05.488229+0000","last_change":"2024-04-18T21:53:01.649275+0000","last_active":"2024-04-18T21:53:05.488229+0000","last_peered":"2024-04-18T21:53:05.488229+0000","last_clean":"2024-04-18T21:53:05.488229+0000","last_became_active":"2024-04-18T21:52:46.664110+0000","last_became_peered":"2024-04-18T21:52:46.664110+0000","last_unstale":"2024-04-18T21:53:05.488229+0000","last_undegraded":"2024-04-18T21:53:05.488229+0000","last_fullsized":"2024-04-18T21:53:05.488229+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:52:49.399707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:49.399707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:05.869842+0000","scrub_duration":1695,"objects_trimmed":0,"snaptrim_duration":0.0011500080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1,6,3,8,4,2],"acting":[10,5,1,6,3,8,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.e","version":"105'80","reported_seq":520,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.383293+0000","last_change":"2024-04-18T21:53:04.093813+0000","last_active":"2024-04-18T21:53:04.383293+0000","last_peered":"2024-04-18T21:53:04.383293+0000","last_clean":"2024-04-18T21:53:04.383293+0000","last_became_active":"2024-04-18T21:52:52.779909+0000","last_became_peered":"2024-04-18T21:52:52.779909+0000","last_unstale":"2024-04-18T21:53:04.383293+0000","last_undegraded":"2024-04-18T21:53:04.383293+0000","last_fullsized":"2024-04-18T21:53:04.383293+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'80","last_scrub_stamp":"2024-04-18T21:53:04.093468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:53:04.093468+0000","objects_scrubbed":4,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:05.844156+0000","scrub_duration":1019,"objects_trimmed":0,"snaptrim_duration":0.50905602100000003,"stat_sum":{"num_bytes":7585792,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":4236,"num_write":36,"num_write_kb":10112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":22839296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1,4,8,11,9,5],"acting":[2,6,1,4,8,11,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.f","version":"101'40","reported_seq":139,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.391125+0000","last_change":"2024-04-18T21:53:01.436409+0000","last_active":"2024-04-18T21:53:04.391125+0000","last_peered":"2024-04-18T21:53:04.391125+0000","last_clean":"2024-04-18T21:53:04.391125+0000","last_became_active":"2024-04-18T21:52:56.896132+0000","last_became_peered":"2024-04-18T21:52:56.896132+0000","last_unstale":"2024-04-18T21:53:04.391125+0000","last_undegraded":"2024-04-18T21:53:04.391125+0000","last_fullsized":"2024-04-18T21:53:04.391125+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:51:41.891788+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001325627,"stat_sum":{"num_bytes":5046272,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":2888,"num_write":43,"num_write_kb":10064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":25657344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,10,4,2,5,11,8],"acting":[3,1,10,4,2,5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.10","version":"109'75","reported_seq":498,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-18T21:53:06.566233+0000","last_change":"2024-04-18T21:53:01.440637+0000","last_active":"2024-04-18T21:53:06.566233+0000","last_peered":"2024-04-18T21:53:06.566233+0000","last_clean":"2024-04-18T21:53:06.566233+0000","last_became_active":"2024-04-18T21:52:52.691769+0000","last_became_peered":"2024-04-18T21:52:52.691769+0000","last_unstale":"2024-04-18T21:53:06.566233+0000","last_undegraded":"2024-04-18T21:53:06.566233+0000","last_fullsized":"2024-04-18T21:53:06.566233+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'16","last_scrub_stamp":"2024-04-18T21:52:03.761423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:03.761423+0000","objects_scrubbed":4,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:17.677199+0000","scrub_duration":763,"objects_trimmed":0,"snaptrim_duration":0.001050076,"stat_sum":{"num_bytes":4472832,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":22,"num_write_kb":7376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":21823488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,11,8,4,10,3],"acting":[6,2,5,11,8,4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.11","version":"98'36","reported_seq":378,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.391337+0000","last_change":"2024-04-18T21:53:01.436541+0000","last_active":"2024-04-18T21:53:04.391337+0000","last_peered":"2024-04-18T21:53:04.391337+0000","last_clean":"2024-04-18T21:53:04.391337+0000","last_became_active":"2024-04-18T21:52:54.949900+0000","last_became_peered":"2024-04-18T21:52:54.949900+0000","last_unstale":"2024-04-18T21:53:04.391337+0000","last_undegraded":"2024-04-18T21:53:04.391337+0000","last_fullsized":"2024-04-18T21:53:04.391337+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'24","last_scrub_stamp":"2024-04-18T21:52:12.154768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:12.154768+0000","objects_scrubbed":5,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:13.516003+0000","scrub_duration":1591,"objects_trimmed":0,"snaptrim_duration":0.0019479040000000001,"stat_sum":{"num_bytes":2637824,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7730,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3899392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6,10,1,5,2,8],"acting":[3,4,6,10,1,5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.12","version":"108'38","reported_seq":420,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.485637+0000","last_change":"2024-04-18T21:53:01.853286+0000","last_active":"2024-04-18T21:53:04.485637+0000","last_peered":"2024-04-18T21:53:04.485637+0000","last_clean":"2024-04-18T21:53:04.485637+0000","last_became_active":"2024-04-18T21:52:57.883159+0000","last_became_peered":"2024-04-18T21:52:57.883159+0000","last_unstale":"2024-04-18T21:53:04.485637+0000","last_undegraded":"2024-04-18T21:53:04.485637+0000","last_fullsized":"2024-04-18T21:53:04.485637+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'11","last_scrub_stamp":"2024-04-18T21:52:15.060778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:15.060778+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:42.886064+0000","scrub_duration":799,"objects_trimmed":0,"snaptrim_duration":0.315364382,"stat_sum":{"num_bytes":4374528,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":3,"num_write":17,"num_write_kb":6096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7372800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8,10,11,5,6,1],"acting":[9,2,8,10,11,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]},{"pgid":"3.13","version":"100'81","reported_seq":417,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-18T21:53:04.382031+0000","last_change":"2024-04-18T21:53:01.441770+0000","last_active":"2024-04-18T21:53:04.382031+0000","last_peered":"2024-04-18T21:53:04.382031+0000","last_clean":"2024-04-18T21:53:04.382031+0000","last_became_active":"2024-04-18T21:52:55.685173+0000","last_became_peered":"2024-04-18T21:52:55.685173+0000","last_unstale":"2024-04-18T21:53:04.382031+0000","last_undegraded":"2024-04-18T21:53:04.382031+0000","last_fullsized":"2024-04-18T21:53:04.382031+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":5,"last_scrub":"68'57","last_scrub_stamp":"2024-04-18T21:52:26.089886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:41.891788+0000","last_clean_scrub_stamp":"2024-04-18T21:52:26.089886+0000","objects_scrubbed":7,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:53:31.357038+0000","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0.0010605090000000001,"stat_sum":{"num_bytes":3096576,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":5110,"num_write":25,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6602752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,8,6,11,1,9],"acting":[2,5,4,8,6,11,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a","length":"1"},{"start":"10","length":"1"},{"start":"15","length":"6"}]}],"pool_stats":[{"poolid":3,"num_pg":22,"stat_sum":{"num_bytes":104005632,"num_objects":58,"num_object_clones":18,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":1,"num_read":313,"num_read_kb":164797,"num_write":562,"num_write_kb":124592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":157712384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207204352,"data_stored":290033664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1045,"ondisk_log_size":1045,"up":176,"acting":176,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378647,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46580,"kb_used_data":11240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325260,"statfs":{"total":96636764160,"available":96589066240,"internally_reserved":0,"allocated":11509760,"data_stored":15990867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378647,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60960,"kb_used_data":25620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94310880,"statfs":{"total":96636764160,"available":96574341120,"internally_reserved":0,"allocated":26234880,"data_stored":37462643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378648,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58020,"kb_used_data":22680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94313820,"statfs":{"total":96636764160,"available":96577351680,"internally_reserved":0,"allocated":23224320,"data_stored":31510611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378647,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57196,"kb_used_data":21856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314644,"statfs":{"total":96636764160,"available":96578195456,"internally_reserved":0,"allocated":22380544,"data_stored":31044211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378647,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55688,"kb_used_data":20340,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316152,"statfs":{"total":96636764160,"available":96579739648,"internally_reserved":0,"allocated":20828160,"data_stored":26791675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,7,1,1,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411350,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53324,"kb_used_data":22136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318516,"statfs":{"total":96636764160,"available":96582160384,"internally_reserved":0,"allocated":22667264,"data_stored":30396155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":0,"seq":77309411350,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411350,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58672,"kb_used_data":23332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94313168,"statfs":{"total":96636764160,"available":96576684032,"internally_reserved":0,"allocated":23891968,"data_stored":31010555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411351,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59344,"kb_used_data":23996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94312496,"statfs":{"total":96636764160,"available":96575995904,"internally_reserved":0,"allocated":24571904,"data_stored":31358715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":77309411349,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444055,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57132,"kb_used_data":21796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314708,"statfs":{"total":96636764160,"available":96578260992,"internally_reserved":0,"allocated":22319104,"data_stored":31802118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444055,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48488,"kb_used_data":17308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323352,"statfs":{"total":96636764160,"available":96587112448,"internally_reserved":0,"allocated":17723392,"data_stored":25330737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3923968,"data_stored":5050368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21254144,"data_stored":30023680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23220224,"data_stored":30543872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19476480,"data_stored":25976832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":22392832,"data_stored":31219712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22540288,"data_stored":30195712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24375296,"data_stored":34713600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":483328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16322560,"data_stored":24539136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":21315584,"data_stored":29581312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":10989568,"data_stored":16117760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20930560,"data_stored":31588352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:53:09.712 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-18T21:53:09.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-18T21:53:09.798 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-04-18T21:53:09.794+0000 7f7127ec8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25093) UID: 0 2024-04-18T21:53:09.844 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:09.844 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:09.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-18T21:53:09.899 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:09.894+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:09.979 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:09.979 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:09.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-18T21:53:10.000 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:09.994+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:10.101 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:10.098+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:10.112 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:10.112 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:10.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-18T21:53:10.218 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-04-18T21:53:10.198+0000 7f86418b5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25115) UID: 0 2024-04-18T21:53:10.290 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:10.290 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:10.302 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:10.298+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:10.305 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-18T21:53:10.403 INFO:tasks.ceph.osd.8.smithi183.stderr:2024-04-18T21:53:10.398+0000 7f8326116640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25333) UID: 0 2024-04-18T21:53:10.474 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:10.474 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:10.482 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:53:10.482 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'unique_pool_0' removed 2024-04-18T21:53:10.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-18T21:53:10.513 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-18T21:53:10.531 INFO:tasks.thrashosds:joining thrashosds 2024-04-18T21:53:10.533 INFO:tasks.ceph.osd.11.smithi183.stderr:2024-04-18T21:53:10.530+0000 7f1645d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25340) UID: 0 2024-04-18T21:53:10.657 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:10.657 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:10.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-18T21:53:10.807 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:10.807 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:10.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-18T21:53:10.949 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:10.949 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:10.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-18T21:53:11.080 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:11.080 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:11.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-18T21:53:11.219 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-04-18T21:53:11.219 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-18T21:53:12.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-18T21:53:13.488 INFO:teuthology.orchestra.run.smithi097.stderr:noscrub is unset 2024-04-18T21:53:13.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-18T21:53:14.502 INFO:teuthology.orchestra.run.smithi097.stderr:nodeep-scrub is unset 2024-04-18T21:53:14.846 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-18T21:53:14.846 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-04-18T21:53:14.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-04-18T21:53:15.160 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-18T21:53:15.154+0000 7f0412e7b640 -1 mon.a@0(leader).osd e116 definitely_dead 0 2024-04-18T21:53:15.505 INFO:teuthology.orchestra.run.smithi097.stderr:marked in osd.7. 2024-04-18T21:53:15.529 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-04-18T21:53:15.529 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-18T21:53:15.529 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-18T21:53:15.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-04-18T21:53:15.849 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-04-18T21:53:15.846+0000 7f0412e7b640 -1 mon.a@0(leader).osd e117 definitely_dead 0 2024-04-18T21:53:16.512 INFO:teuthology.orchestra.run.smithi097.stderr:marked in osd.0. 2024-04-18T21:53:16.537 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-18T21:53:16.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-18T21:53:16.675 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a: {} 2024-04-18T21:53:16.675 INFO:teuthology.orchestra.run.smithi097.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-18T21:53:16.688 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b: {} 2024-04-18T21:53:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-18T21:53:16.702 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c: {} 2024-04-18T21:53:16.703 INFO:teuthology.orchestra.run.smithi097.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-18T21:53:16.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-18T21:53:17.368 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.0 to 1 (10000) 2024-04-18T21:53:17.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-18T21:53:18.371 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-18T21:53:18.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-18T21:53:19.524 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.1 to 1 (10000) 2024-04-18T21:53:19.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-18T21:53:20.546 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-18T21:53:20.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-18T21:53:21.549 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.2 to 1 (10000) 2024-04-18T21:53:21.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-18T21:53:22.558 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-18T21:53:22.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-18T21:53:23.569 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.3 to 1 (10000) 2024-04-18T21:53:23.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-18T21:53:24.577 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-18T21:53:24.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-18T21:53:25.586 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.4 to 1 (10000) 2024-04-18T21:53:25.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-18T21:53:26.595 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-18T21:53:26.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-18T21:53:27.605 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.5 to 1 (10000) 2024-04-18T21:53:27.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-18T21:53:28.613 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-18T21:53:28.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-18T21:53:29.624 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.6 to 1 (10000) 2024-04-18T21:53:29.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-18T21:53:30.634 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-18T21:53:30.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-18T21:53:31.645 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.7 to 1 (10000) 2024-04-18T21:53:31.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-18T21:53:32.656 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-18T21:53:32.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-18T21:53:33.666 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.8 to 1 (10000) 2024-04-18T21:53:33.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-04-18T21:53:34.677 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-18T21:53:34.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-18T21:53:35.689 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.9 to 1 (10000) 2024-04-18T21:53:35.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-04-18T21:53:36.700 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-18T21:53:36.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-18T21:53:37.714 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.10 to 1 (10000) 2024-04-18T21:53:37.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-04-18T21:53:38.724 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-18T21:53:38.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-18T21:53:39.738 INFO:teuthology.orchestra.run.smithi097.stderr:reweighted osd.11 to 1 (10000) 2024-04-18T21:53:39.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-04-18T21:53:40.745 INFO:teuthology.orchestra.run.smithi097.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-18T21:53:40.768 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-18T21:53:40.768 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-18T21:53:40.768 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-18T21:53:40.769 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-18T21:53:40.769 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-18T21:53:40.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T21:53:41.068 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:41.068 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":142,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:53:40.743635+0000","last_up_change":"2024-04-18T21:51:29.769135+0000","last_in_change":"2024-04-18T21:53:16.506431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T21:51:28.321879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-18T21:51:30.835497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.0099999904632568,"score_stable":3.0099999904632568,"optimal_score":0.82999998331069946,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6812","nonce":3458443931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6817","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6820","nonce":3458443931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6831","nonce":3458443931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6828","nonce":3458443931}]},"public_addr":"172.21.15.97:6812/3458443931","cluster_addr":"172.21.15.97:6820/3458443931","heartbeat_back_addr":"172.21.15.97:6831/3458443931","heartbeat_front_addr":"172.21.15.97:6828/3458443931","state":["exists","up"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6801","nonce":1179456160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6803","nonce":1179456160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6811","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6815","nonce":1179456160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6807","nonce":1179456160}]},"public_addr":"172.21.15.97:6801/1179456160","cluster_addr":"172.21.15.97:6803/1179456160","heartbeat_back_addr":"172.21.15.97:6815/1179456160","heartbeat_front_addr":"172.21.15.97:6807/1179456160","state":["exists","up"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6813","nonce":598211345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6819","nonce":598211345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6827","nonce":598211345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6821","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6823","nonce":598211345}]},"public_addr":"172.21.15.97:6813/598211345","cluster_addr":"172.21.15.97:6819/598211345","heartbeat_back_addr":"172.21.15.97:6827/598211345","heartbeat_front_addr":"172.21.15.97:6823/598211345","state":["exists","up"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6806","nonce":40393780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6814","nonce":40393780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6829","nonce":40393780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6822","nonce":40393780}]},"public_addr":"172.21.15.97:6806/40393780","cluster_addr":"172.21.15.97:6814/40393780","heartbeat_back_addr":"172.21.15.97:6829/40393780","heartbeat_front_addr":"172.21.15.97:6822/40393780","state":["exists","up"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6809","nonce":3973913623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6811","nonce":3973913623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6815","nonce":3973913623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6813","nonce":3973913623}]},"public_addr":"172.21.15.104:6809/3973913623","cluster_addr":"172.21.15.104:6811/3973913623","heartbeat_back_addr":"172.21.15.104:6815/3973913623","heartbeat_front_addr":"172.21.15.104:6813/3973913623","state":["exists","up"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6801","nonce":1478992259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6803","nonce":1478992259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6807","nonce":1478992259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6805","nonce":1478992259}]},"public_addr":"172.21.15.104:6801/1478992259","cluster_addr":"172.21.15.104:6803/1478992259","heartbeat_back_addr":"172.21.15.104:6807/1478992259","heartbeat_front_addr":"172.21.15.104:6805/1478992259","state":["exists","up"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6817","nonce":462814955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6819","nonce":462814955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6823","nonce":462814955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6821","nonce":462814955}]},"public_addr":"172.21.15.104:6817/462814955","cluster_addr":"172.21.15.104:6819/462814955","heartbeat_back_addr":"172.21.15.104:6823/462814955","heartbeat_front_addr":"172.21.15.104:6821/462814955","state":["exists","up"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6825","nonce":581913590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6827","nonce":581913590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6831","nonce":581913590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6829","nonce":581913590}]},"public_addr":"172.21.15.104:6825/581913590","cluster_addr":"172.21.15.104:6827/581913590","heartbeat_back_addr":"172.21.15.104:6831/581913590","heartbeat_front_addr":"172.21.15.104:6829/581913590","state":["exists","up"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6806","nonce":1020504720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6812","nonce":1020504720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6825","nonce":1020504720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6819","nonce":1020504720}]},"public_addr":"172.21.15.183:6806/1020504720","cluster_addr":"172.21.15.183:6812/1020504720","heartbeat_back_addr":"172.21.15.183:6825/1020504720","heartbeat_front_addr":"172.21.15.183:6819/1020504720","state":["exists","up"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6814","nonce":1194329140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6821","nonce":1194329140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6829","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6830","nonce":1194329140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6827","nonce":1194329140}]},"public_addr":"172.21.15.183:6814/1194329140","cluster_addr":"172.21.15.183:6821/1194329140","heartbeat_back_addr":"172.21.15.183:6830/1194329140","heartbeat_front_addr":"172.21.15.183:6827/1194329140","state":["exists","up"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6813","nonce":1192340806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6817","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6820","nonce":1192340806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6831","nonce":1192340806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6823","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6826","nonce":1192340806}]},"public_addr":"172.21.15.183:6813/1192340806","cluster_addr":"172.21.15.183:6820/1192340806","heartbeat_back_addr":"172.21.15.183:6831/1192340806","heartbeat_front_addr":"172.21.15.183:6826/1192340806","state":["exists","up"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6801","nonce":3144695911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6803","nonce":3144695911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6815","nonce":3144695911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6807","nonce":3144695911}]},"public_addr":"172.21.15.183:6801/3144695911","cluster_addr":"172.21.15.183:6803/3144695911","heartbeat_back_addr":"172.21.15.183:6815/3144695911","heartbeat_front_addr":"172.21.15.183:6807/3144695911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:32.843393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:55.891649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:56.078049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:45.523740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:35.646618+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:43.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:36.263252+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:53:35.255059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:42.792903+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:38.930203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:35.599364+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:47.013093+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":16,"length":1},{"begin":21,"length":6},{"begin":31,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:53:41.085 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-18T21:53:41.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T21:53:41.379 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":142,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:53:40.743635+0000","last_up_change":"2024-04-18T21:51:29.769135+0000","last_in_change":"2024-04-18T21:53:16.506431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T21:51:28.321879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-18T21:51:30.835497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.0099999904632568,"score_stable":3.0099999904632568,"optimal_score":0.82999998331069946,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6812","nonce":3458443931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6817","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6820","nonce":3458443931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6831","nonce":3458443931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6828","nonce":3458443931}]},"public_addr":"172.21.15.97:6812/3458443931","cluster_addr":"172.21.15.97:6820/3458443931","heartbeat_back_addr":"172.21.15.97:6831/3458443931","heartbeat_front_addr":"172.21.15.97:6828/3458443931","state":["exists","up"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6801","nonce":1179456160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6803","nonce":1179456160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6811","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6815","nonce":1179456160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6807","nonce":1179456160}]},"public_addr":"172.21.15.97:6801/1179456160","cluster_addr":"172.21.15.97:6803/1179456160","heartbeat_back_addr":"172.21.15.97:6815/1179456160","heartbeat_front_addr":"172.21.15.97:6807/1179456160","state":["exists","up"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6813","nonce":598211345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6819","nonce":598211345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6827","nonce":598211345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6821","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6823","nonce":598211345}]},"public_addr":"172.21.15.97:6813/598211345","cluster_addr":"172.21.15.97:6819/598211345","heartbeat_back_addr":"172.21.15.97:6827/598211345","heartbeat_front_addr":"172.21.15.97:6823/598211345","state":["exists","up"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6806","nonce":40393780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6814","nonce":40393780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6829","nonce":40393780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6822","nonce":40393780}]},"public_addr":"172.21.15.97:6806/40393780","cluster_addr":"172.21.15.97:6814/40393780","heartbeat_back_addr":"172.21.15.97:6829/40393780","heartbeat_front_addr":"172.21.15.97:6822/40393780","state":["exists","up"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6809","nonce":3973913623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6811","nonce":3973913623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6815","nonce":3973913623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6813","nonce":3973913623}]},"public_addr":"172.21.15.104:6809/3973913623","cluster_addr":"172.21.15.104:6811/3973913623","heartbeat_back_addr":"172.21.15.104:6815/3973913623","heartbeat_front_addr":"172.21.15.104:6813/3973913623","state":["exists","up"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6801","nonce":1478992259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6803","nonce":1478992259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6807","nonce":1478992259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6805","nonce":1478992259}]},"public_addr":"172.21.15.104:6801/1478992259","cluster_addr":"172.21.15.104:6803/1478992259","heartbeat_back_addr":"172.21.15.104:6807/1478992259","heartbeat_front_addr":"172.21.15.104:6805/1478992259","state":["exists","up"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6817","nonce":462814955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6819","nonce":462814955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6823","nonce":462814955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6821","nonce":462814955}]},"public_addr":"172.21.15.104:6817/462814955","cluster_addr":"172.21.15.104:6819/462814955","heartbeat_back_addr":"172.21.15.104:6823/462814955","heartbeat_front_addr":"172.21.15.104:6821/462814955","state":["exists","up"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6825","nonce":581913590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6827","nonce":581913590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6831","nonce":581913590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6829","nonce":581913590}]},"public_addr":"172.21.15.104:6825/581913590","cluster_addr":"172.21.15.104:6827/581913590","heartbeat_back_addr":"172.21.15.104:6831/581913590","heartbeat_front_addr":"172.21.15.104:6829/581913590","state":["exists","up"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6806","nonce":1020504720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6812","nonce":1020504720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6825","nonce":1020504720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6819","nonce":1020504720}]},"public_addr":"172.21.15.183:6806/1020504720","cluster_addr":"172.21.15.183:6812/1020504720","heartbeat_back_addr":"172.21.15.183:6825/1020504720","heartbeat_front_addr":"172.21.15.183:6819/1020504720","state":["exists","up"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6814","nonce":1194329140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6821","nonce":1194329140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6829","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6830","nonce":1194329140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6827","nonce":1194329140}]},"public_addr":"172.21.15.183:6814/1194329140","cluster_addr":"172.21.15.183:6821/1194329140","heartbeat_back_addr":"172.21.15.183:6830/1194329140","heartbeat_front_addr":"172.21.15.183:6827/1194329140","state":["exists","up"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6813","nonce":1192340806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6817","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6820","nonce":1192340806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6831","nonce":1192340806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6823","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6826","nonce":1192340806}]},"public_addr":"172.21.15.183:6813/1192340806","cluster_addr":"172.21.15.183:6820/1192340806","heartbeat_back_addr":"172.21.15.183:6831/1192340806","heartbeat_front_addr":"172.21.15.183:6826/1192340806","state":["exists","up"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6801","nonce":3144695911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6803","nonce":3144695911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6815","nonce":3144695911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6807","nonce":3144695911}]},"public_addr":"172.21.15.183:6801/3144695911","cluster_addr":"172.21.15.183:6803/3144695911","heartbeat_back_addr":"172.21.15.183:6815/3144695911","heartbeat_front_addr":"172.21.15.183:6807/3144695911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:32.843393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:55.891649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:56.078049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:45.523740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:35.646618+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:43.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:36.263252+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:53:35.255059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:42.792903+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:38.930203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:35.599364+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:47.013093+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":16,"length":1},{"begin":21,"length":6},{"begin":31,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:53:41.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-18T21:53:41.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-18T21:53:41.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-18T21:53:41.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-18T21:53:41.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-18T21:53:41.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-18T21:53:41.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-18T21:53:41.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-18T21:53:41.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-18T21:53:41.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-18T21:53:41.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-18T21:53:41.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-18T21:53:41.730 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:41.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-18T21:53:41.751 INFO:teuthology.orchestra.run.smithi097.stdout:73014444062 2024-04-18T21:53:41.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-18T21:53:41.757 INFO:teuthology.orchestra.run.smithi097.stdout:73014444062 2024-04-18T21:53:41.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-18T21:53:41.758 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:41.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-18T21:53:41.775 INFO:teuthology.orchestra.run.smithi097.stdout:77309411357 2024-04-18T21:53:41.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-18T21:53:41.790 INFO:teuthology.orchestra.run.smithi097.stdout:77309411359 2024-04-18T21:53:41.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-18T21:53:41.803 INFO:teuthology.orchestra.run.smithi097.stdout:77309411358 2024-04-18T21:53:41.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-18T21:53:41.808 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:41.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-18T21:53:41.827 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:41.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-18T21:53:41.829 INFO:teuthology.orchestra.run.smithi097.stdout:77309411358 2024-04-18T21:53:41.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-18T21:53:41.835 INFO:teuthology.orchestra.run.smithi097.stdout:77309411358 2024-04-18T21:53:41.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-18T21:53:41.852 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:41.852 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-18T21:53:42.379 INFO:teuthology.orchestra.run.smithi097.stdout:81604378653 2024-04-18T21:53:42.408 INFO:teuthology.orchestra.run.smithi097.stdout:73014444061 2024-04-18T21:53:42.425 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378653 for osd.1 2024-04-18T21:53:42.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444062 got 73014444061 for osd.8 2024-04-18T21:53:42.509 INFO:teuthology.orchestra.run.smithi097.stdout:77309411358 2024-04-18T21:53:42.522 INFO:teuthology.orchestra.run.smithi097.stdout:81604378654 2024-04-18T21:53:42.532 INFO:teuthology.orchestra.run.smithi097.stdout:81604378653 2024-04-18T21:53:42.545 INFO:teuthology.orchestra.run.smithi097.stdout:77309411357 2024-04-18T21:53:42.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411359 got 77309411358 for osd.2 2024-04-18T21:53:42.550 INFO:teuthology.orchestra.run.smithi097.stdout:81604378653 2024-04-18T21:53:42.557 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378654 for osd.4 2024-04-18T21:53:42.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411358 got 77309411357 for osd.9 2024-04-18T21:53:42.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378653 for osd.6 2024-04-18T21:53:42.583 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378653 for osd.10 2024-04-18T21:53:42.591 INFO:teuthology.orchestra.run.smithi097.stdout:77309411357 2024-04-18T21:53:42.596 INFO:teuthology.orchestra.run.smithi097.stdout:81604378654 2024-04-18T21:53:42.605 INFO:teuthology.orchestra.run.smithi097.stdout:77309411356 2024-04-18T21:53:42.607 INFO:teuthology.orchestra.run.smithi097.stdout:73014444061 2024-04-18T21:53:42.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411358 got 77309411357 for osd.7 2024-04-18T21:53:42.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378654 for osd.3 2024-04-18T21:53:42.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444062 got 73014444061 for osd.11 2024-04-18T21:53:42.630 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411357 got 77309411356 for osd.0 2024-04-18T21:53:42.646 INFO:teuthology.orchestra.run.smithi097.stdout:77309411356 2024-04-18T21:53:42.663 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411358 got 77309411356 for osd.5 2024-04-18T21:53:43.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-18T21:53:43.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-18T21:53:43.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-18T21:53:43.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-18T21:53:43.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-18T21:53:43.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-18T21:53:43.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-18T21:53:43.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-18T21:53:43.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-18T21:53:43.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-18T21:53:43.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-18T21:53:43.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-18T21:53:43.905 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:43.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378655 for osd.1 2024-04-18T21:53:43.958 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:43.968 INFO:teuthology.orchestra.run.smithi097.stdout:73014444062 2024-04-18T21:53:44.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444062 got 73014444062 for osd.8 2024-04-18T21:53:44.007 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.203 INFO:teuthology.orchestra.run.smithi097.stdout:81604378656 2024-04-18T21:53:44.203 INFO:teuthology.orchestra.run.smithi097.stdout:77309411358 2024-04-18T21:53:44.211 INFO:teuthology.orchestra.run.smithi097.stdout:77309411359 2024-04-18T21:53:44.231 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378656 for osd.4 2024-04-18T21:53:44.231 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.253 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411359 got 77309411359 for osd.2 2024-04-18T21:53:44.253 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.264 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411358 got 77309411358 for osd.9 2024-04-18T21:53:44.264 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.283 INFO:teuthology.orchestra.run.smithi097.stdout:77309411358 2024-04-18T21:53:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:44.304 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411358 got 77309411358 for osd.7 2024-04-18T21:53:44.304 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.315 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378655 for osd.10 2024-04-18T21:53:44.316 INFO:teuthology.orchestra.run.smithi097.stdout:77309411357 2024-04-18T21:53:44.316 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.322 INFO:teuthology.orchestra.run.smithi097.stdout:73014444062 2024-04-18T21:53:44.326 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:44.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411357 got 77309411357 for osd.0 2024-04-18T21:53:44.335 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444062 got 73014444062 for osd.11 2024-04-18T21:53:44.343 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378655 for osd.6 2024-04-18T21:53:44.354 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.357 INFO:teuthology.orchestra.run.smithi097.stdout:81604378655 2024-04-18T21:53:44.370 INFO:teuthology.orchestra.run.smithi097.stdout:77309411358 2024-04-18T21:53:44.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378655 got 81604378655 for osd.3 2024-04-18T21:53:44.375 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411358 got 77309411358 for osd.5 2024-04-18T21:53:44.386 DEBUG:teuthology.parallel:result is None 2024-04-18T21:53:44.386 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-18T21:53:44.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:53:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:44.694 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:53:44.710 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-18T21:53:44.289406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":441748,"kb_used_data":21752,"kb_used_omap":18,"kb_used_meta":419757,"kb_avail":1132020332,"statfs":{"total":1159641169920,"available":1159188819968,"internally_reserved":0,"allocated":22274048,"data_stored":13995368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":429831485},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.563941"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":265,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:41.282603+0000","last_change":"2024-04-18T21:53:17.482467+0000","last_active":"2024-04-18T21:53:41.282603+0000","last_peered":"2024-04-18T21:53:41.282603+0000","last_clean":"2024-04-18T21:53:41.282603+0000","last_became_active":"2024-04-18T21:53:17.481925+0000","last_became_peered":"2024-04-18T21:53:17.481925+0000","last_unstale":"2024-04-18T21:53:41.282603+0000","last_undegraded":"2024-04-18T21:53:41.282603+0000","last_fullsized":"2024-04-18T21:53:41.282603+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:01.231382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:01.231382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:01.418478+0000","scrub_duration":567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:40.749983+0000","last_change":"2024-04-18T21:53:19.115464+0000","last_active":"2024-04-18T21:53:40.749983+0000","last_peered":"2024-04-18T21:53:40.749983+0000","last_clean":"2024-04-18T21:53:40.749983+0000","last_became_active":"2024-04-18T21:53:17.485150+0000","last_became_peered":"2024-04-18T21:53:17.485150+0000","last_unstale":"2024-04-18T21:53:40.749983+0000","last_undegraded":"2024-04-18T21:53:40.749983+0000","last_fullsized":"2024-04-18T21:53:40.749983+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_clean_scrub_stamp":"2024-04-18T21:53:19.115220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:29.026663+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":254,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.739018+0000","last_change":"2024-04-18T21:53:03.628642+0000","last_active":"2024-04-18T21:53:39.739018+0000","last_peered":"2024-04-18T21:53:39.739018+0000","last_clean":"2024-04-18T21:53:39.739018+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:53:39.739018+0000","last_undegraded":"2024-04-18T21:53:39.739018+0000","last_fullsized":"2024-04-18T21:53:39.739018+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:03.628514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.628514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:24.839453+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":134,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:43.868596+0000","last_change":"2024-04-18T21:53:00.433742+0000","last_active":"2024-04-18T21:53:43.868596+0000","last_peered":"2024-04-18T21:53:43.868596+0000","last_clean":"2024-04-18T21:53:43.868596+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:53:43.868596+0000","last_undegraded":"2024-04-18T21:53:43.868596+0000","last_fullsized":"2024-04-18T21:53:43.868596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.433398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.433398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.629092+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":255,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.806149+0000","last_change":"2024-04-18T21:52:59.204140+0000","last_active":"2024-04-18T21:53:39.806149+0000","last_peered":"2024-04-18T21:53:39.806149+0000","last_clean":"2024-04-18T21:53:39.806149+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:53:39.806149+0000","last_undegraded":"2024-04-18T21:53:39.806149+0000","last_fullsized":"2024-04-18T21:53:39.806149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:52:59.203898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:52:59.203898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:15.459734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:40.749870+0000","last_change":"2024-04-18T21:53:04.014699+0000","last_active":"2024-04-18T21:53:40.749870+0000","last_peered":"2024-04-18T21:53:40.749870+0000","last_clean":"2024-04-18T21:53:40.749870+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:53:40.749870+0000","last_undegraded":"2024-04-18T21:53:40.749870+0000","last_fullsized":"2024-04-18T21:53:40.749870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:04.014520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:04.014520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:23.188349+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.805883+0000","last_change":"2024-04-18T21:53:16.621779+0000","last_active":"2024-04-18T21:53:39.805883+0000","last_peered":"2024-04-18T21:53:39.805883+0000","last_clean":"2024-04-18T21:53:39.805883+0000","last_became_active":"2024-04-18T21:53:16.620879+0000","last_became_peered":"2024-04-18T21:53:16.620879+0000","last_unstale":"2024-04-18T21:53:39.805883+0000","last_undegraded":"2024-04-18T21:53:39.805883+0000","last_fullsized":"2024-04-18T21:53:39.805883+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:13.569707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:13.569707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:34.692105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":265,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.906233+0000","last_change":"2024-04-18T21:53:00.139630+0000","last_active":"2024-04-18T21:53:39.906233+0000","last_peered":"2024-04-18T21:53:39.906233+0000","last_clean":"2024-04-18T21:53:39.906233+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:53:39.906233+0000","last_undegraded":"2024-04-18T21:53:39.906233+0000","last_fullsized":"2024-04-18T21:53:39.906233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:03.266144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":341,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.905907+0000","last_change":"2024-04-18T21:53:20.655230+0000","last_active":"2024-04-18T21:53:39.905907+0000","last_peered":"2024-04-18T21:53:39.905907+0000","last_clean":"2024-04-18T21:53:39.905907+0000","last_became_active":"2024-04-18T21:53:20.654459+0000","last_became_peered":"2024-04-18T21:53:20.654459+0000","last_unstale":"2024-04-18T21:53:39.905907+0000","last_undegraded":"2024-04-18T21:53:39.905907+0000","last_fullsized":"2024-04-18T21:53:39.905907+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'137","last_scrub_stamp":"2024-04-18T21:53:03.382665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.382665+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:21.791354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378656,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411358,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33424,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338416,"statfs":{"total":96636764160,"available":96602537984,"internally_reserved":0,"allocated":2293760,"data_stored":1603196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411357,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33428,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338412,"statfs":{"total":96636764160,"available":96602533888,"internally_reserved":0,"allocated":2293760,"data_stored":1603196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:53:44.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:53:44.986 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:44.987 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:53:45.002 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-18T21:53:44.289406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":441748,"kb_used_data":21752,"kb_used_omap":18,"kb_used_meta":419757,"kb_avail":1132020332,"statfs":{"total":1159641169920,"available":1159188819968,"internally_reserved":0,"allocated":22274048,"data_stored":13995368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":429831485},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.563941"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":265,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:41.282603+0000","last_change":"2024-04-18T21:53:17.482467+0000","last_active":"2024-04-18T21:53:41.282603+0000","last_peered":"2024-04-18T21:53:41.282603+0000","last_clean":"2024-04-18T21:53:41.282603+0000","last_became_active":"2024-04-18T21:53:17.481925+0000","last_became_peered":"2024-04-18T21:53:17.481925+0000","last_unstale":"2024-04-18T21:53:41.282603+0000","last_undegraded":"2024-04-18T21:53:41.282603+0000","last_fullsized":"2024-04-18T21:53:41.282603+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:01.231382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:01.231382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:01.418478+0000","scrub_duration":567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:40.749983+0000","last_change":"2024-04-18T21:53:19.115464+0000","last_active":"2024-04-18T21:53:40.749983+0000","last_peered":"2024-04-18T21:53:40.749983+0000","last_clean":"2024-04-18T21:53:40.749983+0000","last_became_active":"2024-04-18T21:53:17.485150+0000","last_became_peered":"2024-04-18T21:53:17.485150+0000","last_unstale":"2024-04-18T21:53:40.749983+0000","last_undegraded":"2024-04-18T21:53:40.749983+0000","last_fullsized":"2024-04-18T21:53:40.749983+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_clean_scrub_stamp":"2024-04-18T21:53:19.115220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:29.026663+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":254,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.739018+0000","last_change":"2024-04-18T21:53:03.628642+0000","last_active":"2024-04-18T21:53:39.739018+0000","last_peered":"2024-04-18T21:53:39.739018+0000","last_clean":"2024-04-18T21:53:39.739018+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:53:39.739018+0000","last_undegraded":"2024-04-18T21:53:39.739018+0000","last_fullsized":"2024-04-18T21:53:39.739018+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:03.628514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.628514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:24.839453+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":134,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:43.868596+0000","last_change":"2024-04-18T21:53:00.433742+0000","last_active":"2024-04-18T21:53:43.868596+0000","last_peered":"2024-04-18T21:53:43.868596+0000","last_clean":"2024-04-18T21:53:43.868596+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:53:43.868596+0000","last_undegraded":"2024-04-18T21:53:43.868596+0000","last_fullsized":"2024-04-18T21:53:43.868596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.433398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.433398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.629092+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":255,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.806149+0000","last_change":"2024-04-18T21:52:59.204140+0000","last_active":"2024-04-18T21:53:39.806149+0000","last_peered":"2024-04-18T21:53:39.806149+0000","last_clean":"2024-04-18T21:53:39.806149+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:53:39.806149+0000","last_undegraded":"2024-04-18T21:53:39.806149+0000","last_fullsized":"2024-04-18T21:53:39.806149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:52:59.203898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:52:59.203898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:15.459734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:40.749870+0000","last_change":"2024-04-18T21:53:04.014699+0000","last_active":"2024-04-18T21:53:40.749870+0000","last_peered":"2024-04-18T21:53:40.749870+0000","last_clean":"2024-04-18T21:53:40.749870+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:53:40.749870+0000","last_undegraded":"2024-04-18T21:53:40.749870+0000","last_fullsized":"2024-04-18T21:53:40.749870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:04.014520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:04.014520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:23.188349+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.805883+0000","last_change":"2024-04-18T21:53:16.621779+0000","last_active":"2024-04-18T21:53:39.805883+0000","last_peered":"2024-04-18T21:53:39.805883+0000","last_clean":"2024-04-18T21:53:39.805883+0000","last_became_active":"2024-04-18T21:53:16.620879+0000","last_became_peered":"2024-04-18T21:53:16.620879+0000","last_unstale":"2024-04-18T21:53:39.805883+0000","last_undegraded":"2024-04-18T21:53:39.805883+0000","last_fullsized":"2024-04-18T21:53:39.805883+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:13.569707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:13.569707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:34.692105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":265,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.906233+0000","last_change":"2024-04-18T21:53:00.139630+0000","last_active":"2024-04-18T21:53:39.906233+0000","last_peered":"2024-04-18T21:53:39.906233+0000","last_clean":"2024-04-18T21:53:39.906233+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:53:39.906233+0000","last_undegraded":"2024-04-18T21:53:39.906233+0000","last_fullsized":"2024-04-18T21:53:39.906233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:03.266144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":341,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.905907+0000","last_change":"2024-04-18T21:53:20.655230+0000","last_active":"2024-04-18T21:53:39.905907+0000","last_peered":"2024-04-18T21:53:39.905907+0000","last_clean":"2024-04-18T21:53:39.905907+0000","last_became_active":"2024-04-18T21:53:20.654459+0000","last_became_peered":"2024-04-18T21:53:20.654459+0000","last_unstale":"2024-04-18T21:53:39.905907+0000","last_undegraded":"2024-04-18T21:53:39.905907+0000","last_fullsized":"2024-04-18T21:53:39.905907+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'137","last_scrub_stamp":"2024-04-18T21:53:03.382665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.382665+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:21.791354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378656,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411358,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33424,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338416,"statfs":{"total":96636764160,"available":96602537984,"internally_reserved":0,"allocated":2293760,"data_stored":1603196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411357,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33428,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338412,"statfs":{"total":96636764160,"available":96602533888,"internally_reserved":0,"allocated":2293760,"data_stored":1603196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:53:45.004 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-18T21:53:45.004 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-18T21:53:45.018 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-18T21:53:45.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:53:45.285 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:45.285 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:53:45.301 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-18T21:53:44.289406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":441748,"kb_used_data":21752,"kb_used_omap":18,"kb_used_meta":419757,"kb_avail":1132020332,"statfs":{"total":1159641169920,"available":1159188819968,"internally_reserved":0,"allocated":22274048,"data_stored":13995368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":429831485},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.563941"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":265,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:41.282603+0000","last_change":"2024-04-18T21:53:17.482467+0000","last_active":"2024-04-18T21:53:41.282603+0000","last_peered":"2024-04-18T21:53:41.282603+0000","last_clean":"2024-04-18T21:53:41.282603+0000","last_became_active":"2024-04-18T21:53:17.481925+0000","last_became_peered":"2024-04-18T21:53:17.481925+0000","last_unstale":"2024-04-18T21:53:41.282603+0000","last_undegraded":"2024-04-18T21:53:41.282603+0000","last_fullsized":"2024-04-18T21:53:41.282603+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:01.231382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:01.231382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:01.418478+0000","scrub_duration":567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:40.749983+0000","last_change":"2024-04-18T21:53:19.115464+0000","last_active":"2024-04-18T21:53:40.749983+0000","last_peered":"2024-04-18T21:53:40.749983+0000","last_clean":"2024-04-18T21:53:40.749983+0000","last_became_active":"2024-04-18T21:53:17.485150+0000","last_became_peered":"2024-04-18T21:53:17.485150+0000","last_unstale":"2024-04-18T21:53:40.749983+0000","last_undegraded":"2024-04-18T21:53:40.749983+0000","last_fullsized":"2024-04-18T21:53:40.749983+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_clean_scrub_stamp":"2024-04-18T21:53:19.115220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:29.026663+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":254,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.739018+0000","last_change":"2024-04-18T21:53:03.628642+0000","last_active":"2024-04-18T21:53:39.739018+0000","last_peered":"2024-04-18T21:53:39.739018+0000","last_clean":"2024-04-18T21:53:39.739018+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:53:39.739018+0000","last_undegraded":"2024-04-18T21:53:39.739018+0000","last_fullsized":"2024-04-18T21:53:39.739018+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:03.628514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.628514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:24.839453+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":134,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:43.868596+0000","last_change":"2024-04-18T21:53:00.433742+0000","last_active":"2024-04-18T21:53:43.868596+0000","last_peered":"2024-04-18T21:53:43.868596+0000","last_clean":"2024-04-18T21:53:43.868596+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:53:43.868596+0000","last_undegraded":"2024-04-18T21:53:43.868596+0000","last_fullsized":"2024-04-18T21:53:43.868596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.433398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.433398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.629092+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":255,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.806149+0000","last_change":"2024-04-18T21:52:59.204140+0000","last_active":"2024-04-18T21:53:39.806149+0000","last_peered":"2024-04-18T21:53:39.806149+0000","last_clean":"2024-04-18T21:53:39.806149+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:53:39.806149+0000","last_undegraded":"2024-04-18T21:53:39.806149+0000","last_fullsized":"2024-04-18T21:53:39.806149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:52:59.203898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:52:59.203898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:15.459734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:40.749870+0000","last_change":"2024-04-18T21:53:04.014699+0000","last_active":"2024-04-18T21:53:40.749870+0000","last_peered":"2024-04-18T21:53:40.749870+0000","last_clean":"2024-04-18T21:53:40.749870+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:53:40.749870+0000","last_undegraded":"2024-04-18T21:53:40.749870+0000","last_fullsized":"2024-04-18T21:53:40.749870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:04.014520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:04.014520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:23.188349+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.805883+0000","last_change":"2024-04-18T21:53:16.621779+0000","last_active":"2024-04-18T21:53:39.805883+0000","last_peered":"2024-04-18T21:53:39.805883+0000","last_clean":"2024-04-18T21:53:39.805883+0000","last_became_active":"2024-04-18T21:53:16.620879+0000","last_became_peered":"2024-04-18T21:53:16.620879+0000","last_unstale":"2024-04-18T21:53:39.805883+0000","last_undegraded":"2024-04-18T21:53:39.805883+0000","last_fullsized":"2024-04-18T21:53:39.805883+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:13.569707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:13.569707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:34.692105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":265,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.906233+0000","last_change":"2024-04-18T21:53:00.139630+0000","last_active":"2024-04-18T21:53:39.906233+0000","last_peered":"2024-04-18T21:53:39.906233+0000","last_clean":"2024-04-18T21:53:39.906233+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:53:39.906233+0000","last_undegraded":"2024-04-18T21:53:39.906233+0000","last_fullsized":"2024-04-18T21:53:39.906233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:03.266144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":341,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.905907+0000","last_change":"2024-04-18T21:53:20.655230+0000","last_active":"2024-04-18T21:53:39.905907+0000","last_peered":"2024-04-18T21:53:39.905907+0000","last_clean":"2024-04-18T21:53:39.905907+0000","last_became_active":"2024-04-18T21:53:20.654459+0000","last_became_peered":"2024-04-18T21:53:20.654459+0000","last_unstale":"2024-04-18T21:53:39.905907+0000","last_undegraded":"2024-04-18T21:53:39.905907+0000","last_fullsized":"2024-04-18T21:53:39.905907+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'137","last_scrub_stamp":"2024-04-18T21:53:03.382665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.382665+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:21.791354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378656,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411358,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33424,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338416,"statfs":{"total":96636764160,"available":96602537984,"internally_reserved":0,"allocated":2293760,"data_stored":1603196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411357,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33428,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338412,"statfs":{"total":96636764160,"available":96602533888,"internally_reserved":0,"allocated":2293760,"data_stored":1603196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:53:45.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:53:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:53:45.589 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-18T21:53:44.289406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":441748,"kb_used_data":21752,"kb_used_omap":18,"kb_used_meta":419757,"kb_avail":1132020332,"statfs":{"total":1159641169920,"available":1159188819968,"internally_reserved":0,"allocated":22274048,"data_stored":13995368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":429831485},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.563941"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":265,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:41.282603+0000","last_change":"2024-04-18T21:53:17.482467+0000","last_active":"2024-04-18T21:53:41.282603+0000","last_peered":"2024-04-18T21:53:41.282603+0000","last_clean":"2024-04-18T21:53:41.282603+0000","last_became_active":"2024-04-18T21:53:17.481925+0000","last_became_peered":"2024-04-18T21:53:17.481925+0000","last_unstale":"2024-04-18T21:53:41.282603+0000","last_undegraded":"2024-04-18T21:53:41.282603+0000","last_fullsized":"2024-04-18T21:53:41.282603+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:01.231382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:01.231382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:01.418478+0000","scrub_duration":567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:40.749983+0000","last_change":"2024-04-18T21:53:19.115464+0000","last_active":"2024-04-18T21:53:40.749983+0000","last_peered":"2024-04-18T21:53:40.749983+0000","last_clean":"2024-04-18T21:53:40.749983+0000","last_became_active":"2024-04-18T21:53:17.485150+0000","last_became_peered":"2024-04-18T21:53:17.485150+0000","last_unstale":"2024-04-18T21:53:40.749983+0000","last_undegraded":"2024-04-18T21:53:40.749983+0000","last_fullsized":"2024-04-18T21:53:40.749983+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_clean_scrub_stamp":"2024-04-18T21:53:19.115220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:29.026663+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":254,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.739018+0000","last_change":"2024-04-18T21:53:03.628642+0000","last_active":"2024-04-18T21:53:39.739018+0000","last_peered":"2024-04-18T21:53:39.739018+0000","last_clean":"2024-04-18T21:53:39.739018+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:53:39.739018+0000","last_undegraded":"2024-04-18T21:53:39.739018+0000","last_fullsized":"2024-04-18T21:53:39.739018+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:03.628514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.628514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:24.839453+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":134,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:43.868596+0000","last_change":"2024-04-18T21:53:00.433742+0000","last_active":"2024-04-18T21:53:43.868596+0000","last_peered":"2024-04-18T21:53:43.868596+0000","last_clean":"2024-04-18T21:53:43.868596+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:53:43.868596+0000","last_undegraded":"2024-04-18T21:53:43.868596+0000","last_fullsized":"2024-04-18T21:53:43.868596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.433398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.433398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.629092+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":255,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.806149+0000","last_change":"2024-04-18T21:52:59.204140+0000","last_active":"2024-04-18T21:53:39.806149+0000","last_peered":"2024-04-18T21:53:39.806149+0000","last_clean":"2024-04-18T21:53:39.806149+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:53:39.806149+0000","last_undegraded":"2024-04-18T21:53:39.806149+0000","last_fullsized":"2024-04-18T21:53:39.806149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:52:59.203898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:52:59.203898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:15.459734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:40.749870+0000","last_change":"2024-04-18T21:53:04.014699+0000","last_active":"2024-04-18T21:53:40.749870+0000","last_peered":"2024-04-18T21:53:40.749870+0000","last_clean":"2024-04-18T21:53:40.749870+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:53:40.749870+0000","last_undegraded":"2024-04-18T21:53:40.749870+0000","last_fullsized":"2024-04-18T21:53:40.749870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:04.014520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:04.014520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:23.188349+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.805883+0000","last_change":"2024-04-18T21:53:16.621779+0000","last_active":"2024-04-18T21:53:39.805883+0000","last_peered":"2024-04-18T21:53:39.805883+0000","last_clean":"2024-04-18T21:53:39.805883+0000","last_became_active":"2024-04-18T21:53:16.620879+0000","last_became_peered":"2024-04-18T21:53:16.620879+0000","last_unstale":"2024-04-18T21:53:39.805883+0000","last_undegraded":"2024-04-18T21:53:39.805883+0000","last_fullsized":"2024-04-18T21:53:39.805883+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:13.569707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:13.569707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:34.692105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":265,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.906233+0000","last_change":"2024-04-18T21:53:00.139630+0000","last_active":"2024-04-18T21:53:39.906233+0000","last_peered":"2024-04-18T21:53:39.906233+0000","last_clean":"2024-04-18T21:53:39.906233+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:53:39.906233+0000","last_undegraded":"2024-04-18T21:53:39.906233+0000","last_fullsized":"2024-04-18T21:53:39.906233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:03.266144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":341,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-04-18T21:53:39.905907+0000","last_change":"2024-04-18T21:53:20.655230+0000","last_active":"2024-04-18T21:53:39.905907+0000","last_peered":"2024-04-18T21:53:39.905907+0000","last_clean":"2024-04-18T21:53:39.905907+0000","last_became_active":"2024-04-18T21:53:20.654459+0000","last_became_peered":"2024-04-18T21:53:20.654459+0000","last_unstale":"2024-04-18T21:53:39.905907+0000","last_undegraded":"2024-04-18T21:53:39.905907+0000","last_fullsized":"2024-04-18T21:53:39.905907+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'137","last_scrub_stamp":"2024-04-18T21:53:03.382665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.382665+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:21.791354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378656,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411358,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33424,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338416,"statfs":{"total":96636764160,"available":96602537984,"internally_reserved":0,"allocated":2293760,"data_stored":1603196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411357,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33428,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338412,"statfs":{"total":96636764160,"available":96602533888,"internally_reserved":0,"allocated":2293760,"data_stored":1603196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:53:45.590 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-18T21:53:45.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:53:45.871 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:53:45.888 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-04-18T21:53:45.771978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":441816,"kb_used_data":21848,"kb_used_omap":18,"kb_used_meta":419757,"kb_avail":1132020264,"statfs":{"total":1159641169920,"available":1159188750336,"internally_reserved":0,"allocated":22372352,"data_stored":14057222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":429831485},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.035506"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":265,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-18T21:53:41.282603+0000","last_change":"2024-04-18T21:53:17.482467+0000","last_active":"2024-04-18T21:53:41.282603+0000","last_peered":"2024-04-18T21:53:41.282603+0000","last_clean":"2024-04-18T21:53:41.282603+0000","last_became_active":"2024-04-18T21:53:17.481925+0000","last_became_peered":"2024-04-18T21:53:17.481925+0000","last_unstale":"2024-04-18T21:53:41.282603+0000","last_undegraded":"2024-04-18T21:53:41.282603+0000","last_fullsized":"2024-04-18T21:53:41.282603+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:01.231382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:01.231382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:01.418478+0000","scrub_duration":567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":268,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:44.622608+0000","last_change":"2024-04-18T21:53:19.115464+0000","last_active":"2024-04-18T21:53:44.622608+0000","last_peered":"2024-04-18T21:53:44.622608+0000","last_clean":"2024-04-18T21:53:44.622608+0000","last_became_active":"2024-04-18T21:53:17.485150+0000","last_became_peered":"2024-04-18T21:53:17.485150+0000","last_unstale":"2024-04-18T21:53:44.622608+0000","last_undegraded":"2024-04-18T21:53:44.622608+0000","last_fullsized":"2024-04-18T21:53:44.622608+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_clean_scrub_stamp":"2024-04-18T21:53:19.115220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:29.026663+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":258,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:43.865420+0000","last_change":"2024-04-18T21:53:03.628642+0000","last_active":"2024-04-18T21:53:43.865420+0000","last_peered":"2024-04-18T21:53:43.865420+0000","last_clean":"2024-04-18T21:53:43.865420+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:53:43.865420+0000","last_undegraded":"2024-04-18T21:53:43.865420+0000","last_fullsized":"2024-04-18T21:53:43.865420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:03.628514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.628514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:24.839453+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":134,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:43.868596+0000","last_change":"2024-04-18T21:53:00.433742+0000","last_active":"2024-04-18T21:53:43.868596+0000","last_peered":"2024-04-18T21:53:43.868596+0000","last_clean":"2024-04-18T21:53:43.868596+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:53:43.868596+0000","last_undegraded":"2024-04-18T21:53:43.868596+0000","last_fullsized":"2024-04-18T21:53:43.868596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.433398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.433398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.629092+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":259,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:43.765612+0000","last_change":"2024-04-18T21:52:59.204140+0000","last_active":"2024-04-18T21:53:43.765612+0000","last_peered":"2024-04-18T21:53:43.765612+0000","last_clean":"2024-04-18T21:53:43.765612+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:53:43.765612+0000","last_undegraded":"2024-04-18T21:53:43.765612+0000","last_fullsized":"2024-04-18T21:53:43.765612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:52:59.203898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:52:59.203898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:15.459734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":268,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:44.622484+0000","last_change":"2024-04-18T21:53:04.014699+0000","last_active":"2024-04-18T21:53:44.622484+0000","last_peered":"2024-04-18T21:53:44.622484+0000","last_clean":"2024-04-18T21:53:44.622484+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:53:44.622484+0000","last_undegraded":"2024-04-18T21:53:44.622484+0000","last_fullsized":"2024-04-18T21:53:44.622484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:04.014520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:04.014520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:23.188349+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":57,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:45.018465+0000","last_change":"2024-04-18T21:53:16.621779+0000","last_active":"2024-04-18T21:53:45.018465+0000","last_peered":"2024-04-18T21:53:45.018465+0000","last_clean":"2024-04-18T21:53:45.018465+0000","last_became_active":"2024-04-18T21:53:16.620879+0000","last_became_peered":"2024-04-18T21:53:16.620879+0000","last_unstale":"2024-04-18T21:53:45.018465+0000","last_undegraded":"2024-04-18T21:53:45.018465+0000","last_fullsized":"2024-04-18T21:53:45.018465+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:13.569707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:13.569707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:34.692105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":269,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:43.765650+0000","last_change":"2024-04-18T21:53:00.139630+0000","last_active":"2024-04-18T21:53:43.765650+0000","last_peered":"2024-04-18T21:53:43.765650+0000","last_clean":"2024-04-18T21:53:43.765650+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:53:43.765650+0000","last_undegraded":"2024-04-18T21:53:43.765650+0000","last_fullsized":"2024-04-18T21:53:43.765650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:03.266144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":345,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-18T21:53:45.018199+0000","last_change":"2024-04-18T21:53:20.655230+0000","last_active":"2024-04-18T21:53:45.018199+0000","last_peered":"2024-04-18T21:53:45.018199+0000","last_clean":"2024-04-18T21:53:45.018199+0000","last_became_active":"2024-04-18T21:53:20.654459+0000","last_became_peered":"2024-04-18T21:53:20.654459+0000","last_unstale":"2024-04-18T21:53:45.018199+0000","last_undegraded":"2024-04-18T21:53:45.018199+0000","last_fullsized":"2024-04-18T21:53:45.018199+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'137","last_scrub_stamp":"2024-04-18T21:53:03.382665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.382665+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:21.791354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1761280,"data_stored":1074268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378656,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378656,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1785856,"data_stored":1089742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411359,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33432,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338408,"statfs":{"total":96636764160,"available":96602529792,"internally_reserved":0,"allocated":2306048,"data_stored":1610912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41244,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1785856,"data_stored":1089742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411360,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37084,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1785856,"data_stored":1089742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411358,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33432,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338408,"statfs":{"total":96636764160,"available":96602529792,"internally_reserved":0,"allocated":2306048,"data_stored":1610912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":1081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:53:45.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T21:53:46.163 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:46.163 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":144,"fsid":"9a7b119f-53a7-4e46-8d6e-229eac84ebee","created":"2024-04-18T21:51:17.630312+0000","modified":"2024-04-18T21:53:45.769334+0000","last_up_change":"2024-04-18T21:51:29.769135+0000","last_in_change":"2024-04-18T21:53:16.506431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T21:51:28.321879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-18T21:51:30.835497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.0099999904632568,"score_stable":3.0099999904632568,"optimal_score":0.82999998331069946,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a27387f-73d5-4bf2-bb86-80c069a1d6f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6812","nonce":3458443931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6817","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6820","nonce":3458443931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6831","nonce":3458443931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3458443931},{"type":"v1","addr":"172.21.15.97:6828","nonce":3458443931}]},"public_addr":"172.21.15.97:6812/3458443931","cluster_addr":"172.21.15.97:6820/3458443931","heartbeat_back_addr":"172.21.15.97:6831/3458443931","heartbeat_front_addr":"172.21.15.97:6828/3458443931","state":["exists","up"]},{"osd":1,"uuid":"2f6c823f-4b24-4ecf-9561-710723acded2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6801","nonce":1179456160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6803","nonce":1179456160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6811","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6815","nonce":1179456160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6805","nonce":1179456160},{"type":"v1","addr":"172.21.15.97:6807","nonce":1179456160}]},"public_addr":"172.21.15.97:6801/1179456160","cluster_addr":"172.21.15.97:6803/1179456160","heartbeat_back_addr":"172.21.15.97:6815/1179456160","heartbeat_front_addr":"172.21.15.97:6807/1179456160","state":["exists","up"]},{"osd":2,"uuid":"05866927-b7ea-44e6-8a5e-f23263011fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6809","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6813","nonce":598211345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6819","nonce":598211345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6825","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6827","nonce":598211345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6821","nonce":598211345},{"type":"v1","addr":"172.21.15.97:6823","nonce":598211345}]},"public_addr":"172.21.15.97:6813/598211345","cluster_addr":"172.21.15.97:6819/598211345","heartbeat_back_addr":"172.21.15.97:6827/598211345","heartbeat_front_addr":"172.21.15.97:6823/598211345","state":["exists","up"]},{"osd":3,"uuid":"1fbdd117-432a-4bef-879e-96823b61820a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6806","nonce":40393780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6814","nonce":40393780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6829","nonce":40393780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":40393780},{"type":"v1","addr":"172.21.15.97:6822","nonce":40393780}]},"public_addr":"172.21.15.97:6806/40393780","cluster_addr":"172.21.15.97:6814/40393780","heartbeat_back_addr":"172.21.15.97:6829/40393780","heartbeat_front_addr":"172.21.15.97:6822/40393780","state":["exists","up"]},{"osd":4,"uuid":"a7f8da99-45f2-4d8f-8c8c-efd9632b0f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6809","nonce":3973913623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6811","nonce":3973913623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6815","nonce":3973913623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3973913623},{"type":"v1","addr":"172.21.15.104:6813","nonce":3973913623}]},"public_addr":"172.21.15.104:6809/3973913623","cluster_addr":"172.21.15.104:6811/3973913623","heartbeat_back_addr":"172.21.15.104:6815/3973913623","heartbeat_front_addr":"172.21.15.104:6813/3973913623","state":["exists","up"]},{"osd":5,"uuid":"2230dfaa-7c85-469b-a95b-594960d77b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6801","nonce":1478992259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6803","nonce":1478992259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6807","nonce":1478992259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1478992259},{"type":"v1","addr":"172.21.15.104:6805","nonce":1478992259}]},"public_addr":"172.21.15.104:6801/1478992259","cluster_addr":"172.21.15.104:6803/1478992259","heartbeat_back_addr":"172.21.15.104:6807/1478992259","heartbeat_front_addr":"172.21.15.104:6805/1478992259","state":["exists","up"]},{"osd":6,"uuid":"e7437af9-b161-482e-8939-4d2b4c2da10e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6817","nonce":462814955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6819","nonce":462814955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6823","nonce":462814955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":462814955},{"type":"v1","addr":"172.21.15.104:6821","nonce":462814955}]},"public_addr":"172.21.15.104:6817/462814955","cluster_addr":"172.21.15.104:6819/462814955","heartbeat_back_addr":"172.21.15.104:6823/462814955","heartbeat_front_addr":"172.21.15.104:6821/462814955","state":["exists","up"]},{"osd":7,"uuid":"a062c3b3-7e00-4bdf-9731-0072826522c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6825","nonce":581913590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6827","nonce":581913590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6831","nonce":581913590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":581913590},{"type":"v1","addr":"172.21.15.104:6829","nonce":581913590}]},"public_addr":"172.21.15.104:6825/581913590","cluster_addr":"172.21.15.104:6827/581913590","heartbeat_back_addr":"172.21.15.104:6831/581913590","heartbeat_front_addr":"172.21.15.104:6829/581913590","state":["exists","up"]},{"osd":8,"uuid":"f2473c34-2db3-4806-a6ea-bb46f42c4ab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6806","nonce":1020504720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6812","nonce":1020504720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6825","nonce":1020504720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1020504720},{"type":"v1","addr":"172.21.15.183:6819","nonce":1020504720}]},"public_addr":"172.21.15.183:6806/1020504720","cluster_addr":"172.21.15.183:6812/1020504720","heartbeat_back_addr":"172.21.15.183:6825/1020504720","heartbeat_front_addr":"172.21.15.183:6819/1020504720","state":["exists","up"]},{"osd":9,"uuid":"f08d755f-cc20-468c-b3ca-cfb88b44bdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6814","nonce":1194329140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6821","nonce":1194329140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6829","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6830","nonce":1194329140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1194329140},{"type":"v1","addr":"172.21.15.183:6827","nonce":1194329140}]},"public_addr":"172.21.15.183:6814/1194329140","cluster_addr":"172.21.15.183:6821/1194329140","heartbeat_back_addr":"172.21.15.183:6830/1194329140","heartbeat_front_addr":"172.21.15.183:6827/1194329140","state":["exists","up"]},{"osd":10,"uuid":"247fad9b-8847-416f-bfbb-9d8b466c3aa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6809","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6813","nonce":1192340806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6817","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6820","nonce":1192340806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6831","nonce":1192340806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6823","nonce":1192340806},{"type":"v1","addr":"172.21.15.183:6826","nonce":1192340806}]},"public_addr":"172.21.15.183:6813/1192340806","cluster_addr":"172.21.15.183:6820/1192340806","heartbeat_back_addr":"172.21.15.183:6831/1192340806","heartbeat_front_addr":"172.21.15.183:6826/1192340806","state":["exists","up"]},{"osd":11,"uuid":"4a51c1be-4125-451a-a910-c8e364fe21be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6801","nonce":3144695911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6803","nonce":3144695911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6815","nonce":3144695911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":3144695911},{"type":"v1","addr":"172.21.15.183:6807","nonce":3144695911}]},"public_addr":"172.21.15.183:6801/3144695911","cluster_addr":"172.21.15.183:6803/3144695911","heartbeat_back_addr":"172.21.15.183:6815/3144695911","heartbeat_front_addr":"172.21.15.183:6807/3144695911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:53:43.164451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:55.891649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:56.078049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:45.523740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:35.646618+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:43.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:36.263252+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:53:35.255059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:53:44.764146+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:38.930203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:35.599364+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T21:52:47.013093+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":16,"length":1},{"begin":21,"length":6},{"begin":31,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T21:53:47.180 INFO:tasks.ceph:Scrubbing osd.0 2024-04-18T21:53:47.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:47.341 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:47.341 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:47.341 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:47.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-18T21:53:47.640 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 0 to deep-scrub 2024-04-18T21:53:47.658 INFO:tasks.ceph:Scrubbing osd.1 2024-04-18T21:53:47.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:47.811 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:47.811 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:47.811 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:47.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-18T21:53:48.115 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 1 to deep-scrub 2024-04-18T21:53:48.134 INFO:tasks.ceph:Scrubbing osd.2 2024-04-18T21:53:48.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:48.276 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:48.276 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:48.276 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:48.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-18T21:53:48.566 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 2 to deep-scrub 2024-04-18T21:53:48.585 INFO:tasks.ceph:Scrubbing osd.3 2024-04-18T21:53:48.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:48.728 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:48.728 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:48.728 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:48.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-18T21:53:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 3 to deep-scrub 2024-04-18T21:53:49.036 INFO:tasks.ceph:Scrubbing osd.4 2024-04-18T21:53:49.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:49.186 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:49.187 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:49.187 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:49.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-18T21:53:49.486 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 4 to deep-scrub 2024-04-18T21:53:49.504 INFO:tasks.ceph:Scrubbing osd.5 2024-04-18T21:53:49.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:49.639 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:49.639 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:49.639 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:49.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-18T21:53:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 5 to deep-scrub 2024-04-18T21:53:49.942 INFO:tasks.ceph:Scrubbing osd.6 2024-04-18T21:53:49.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:50.097 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:50.097 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:50.097 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:50.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-18T21:53:50.406 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 6 to deep-scrub 2024-04-18T21:53:50.423 INFO:tasks.ceph:Scrubbing osd.7 2024-04-18T21:53:50.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:50.566 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:50.566 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:50.566 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:50.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-18T21:53:50.847 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 7 to deep-scrub 2024-04-18T21:53:50.865 INFO:tasks.ceph:Scrubbing osd.8 2024-04-18T21:53:50.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:51.015 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:51.016 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:51.016 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:51.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-04-18T21:53:51.322 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 8 to deep-scrub 2024-04-18T21:53:51.340 INFO:tasks.ceph:Scrubbing osd.9 2024-04-18T21:53:51.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:51.491 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:51.491 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:51.491 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:51.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-04-18T21:53:51.786 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 9 to deep-scrub 2024-04-18T21:53:51.804 INFO:tasks.ceph:Scrubbing osd.10 2024-04-18T21:53:51.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:51.938 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:51.938 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:51.938 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:51.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-04-18T21:53:52.218 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 10 to deep-scrub 2024-04-18T21:53:52.236 INFO:tasks.ceph:Scrubbing osd.11 2024-04-18T21:53:52.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-04-18T21:53:52.374 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-18T21:53:52.374 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T21:53:52.374 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-18T21:53:52.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-04-18T21:53:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 11 to deep-scrub 2024-04-18T21:53:52.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:53:52.971 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:53:52.972 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:53:52.990 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-04-18T21:53:52.812118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":442060,"kb_used_data":22112,"kb_used_omap":18,"kb_used_meta":419757,"kb_avail":1132020020,"statfs":{"total":1159641169920,"available":1159188500480,"internally_reserved":0,"allocated":22642688,"data_stored":14227856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":429831485},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.522711"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":278,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:50.968891+0000","last_change":"2024-04-18T21:53:50.968949+0000","last_active":"2024-04-18T21:53:50.968891+0000","last_peered":"2024-04-18T21:53:50.968891+0000","last_clean":"2024-04-18T21:53:50.968891+0000","last_became_active":"2024-04-18T21:53:17.481925+0000","last_became_peered":"2024-04-18T21:53:17.481925+0000","last_unstale":"2024-04-18T21:53:50.968891+0000","last_undegraded":"2024-04-18T21:53:50.968891+0000","last_fullsized":"2024-04-18T21:53:50.968891+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:50.968699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:50.968699+0000","last_clean_scrub_stamp":"2024-04-18T21:53:50.968699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:52.238958+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":270,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:45.777708+0000","last_change":"2024-04-18T21:53:19.115464+0000","last_active":"2024-04-18T21:53:45.777708+0000","last_peered":"2024-04-18T21:53:45.777708+0000","last_clean":"2024-04-18T21:53:45.777708+0000","last_became_active":"2024-04-18T21:53:17.485150+0000","last_became_peered":"2024-04-18T21:53:17.485150+0000","last_unstale":"2024-04-18T21:53:45.777708+0000","last_undegraded":"2024-04-18T21:53:45.777708+0000","last_fullsized":"2024-04-18T21:53:45.777708+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:19.115220+0000","last_clean_scrub_stamp":"2024-04-18T21:53:19.115220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:29.026663+0000","scrub_duration":407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:49.674252+0000","last_change":"2024-04-18T21:53:49.674275+0000","last_active":"2024-04-18T21:53:49.674252+0000","last_peered":"2024-04-18T21:53:49.674252+0000","last_clean":"2024-04-18T21:53:49.674252+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:53:49.674252+0000","last_undegraded":"2024-04-18T21:53:49.674252+0000","last_fullsized":"2024-04-18T21:53:49.674252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:49.674156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:49.674156+0000","last_clean_scrub_stamp":"2024-04-18T21:53:49.674156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:16.819592+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:46.839643+0000","last_change":"2024-04-18T21:53:00.433742+0000","last_active":"2024-04-18T21:53:46.839643+0000","last_peered":"2024-04-18T21:53:46.839643+0000","last_clean":"2024-04-18T21:53:46.839643+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:53:46.839643+0000","last_undegraded":"2024-04-18T21:53:46.839643+0000","last_fullsized":"2024-04-18T21:53:46.839643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.433398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.433398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.629092+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":262,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:46.393589+0000","last_change":"2024-04-18T21:52:59.204140+0000","last_active":"2024-04-18T21:53:46.393589+0000","last_peered":"2024-04-18T21:53:46.393589+0000","last_clean":"2024-04-18T21:53:46.393589+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:53:46.393589+0000","last_undegraded":"2024-04-18T21:53:46.393589+0000","last_fullsized":"2024-04-18T21:53:46.393589+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:52:59.203898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:52:59.203898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":270,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:45.777689+0000","last_change":"2024-04-18T21:53:04.014699+0000","last_active":"2024-04-18T21:53:45.777689+0000","last_peered":"2024-04-18T21:53:45.777689+0000","last_clean":"2024-04-18T21:53:45.777689+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:53:45.777689+0000","last_undegraded":"2024-04-18T21:53:45.777689+0000","last_fullsized":"2024-04-18T21:53:45.777689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:04.014520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:04.014520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:23.188349+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":59,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:46.839188+0000","last_change":"2024-04-18T21:53:16.621779+0000","last_active":"2024-04-18T21:53:46.839188+0000","last_peered":"2024-04-18T21:53:46.839188+0000","last_clean":"2024-04-18T21:53:46.839188+0000","last_became_active":"2024-04-18T21:53:16.620879+0000","last_became_peered":"2024-04-18T21:53:16.620879+0000","last_unstale":"2024-04-18T21:53:46.839188+0000","last_undegraded":"2024-04-18T21:53:46.839188+0000","last_fullsized":"2024-04-18T21:53:46.839188+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:13.569707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:13.569707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:34.692105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":272,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:46.393578+0000","last_change":"2024-04-18T21:53:00.139630+0000","last_active":"2024-04-18T21:53:46.393578+0000","last_peered":"2024-04-18T21:53:46.393578+0000","last_clean":"2024-04-18T21:53:46.393578+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:53:46.393578+0000","last_undegraded":"2024-04-18T21:53:46.393578+0000","last_fullsized":"2024-04-18T21:53:46.393578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:00.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:31.787097+0000","last_clean_scrub_stamp":"2024-04-18T21:53:00.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"28'137","reported_seq":347,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-18T21:53:46.738999+0000","last_change":"2024-04-18T21:53:20.655230+0000","last_active":"2024-04-18T21:53:46.738999+0000","last_peered":"2024-04-18T21:53:46.738999+0000","last_clean":"2024-04-18T21:53:46.738999+0000","last_became_active":"2024-04-18T21:53:20.654459+0000","last_became_peered":"2024-04-18T21:53:20.654459+0000","last_unstale":"2024-04-18T21:53:46.738999+0000","last_undegraded":"2024-04-18T21:53:46.738999+0000","last_fullsized":"2024-04-18T21:53:46.738999+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'137","last_scrub_stamp":"2024-04-18T21:53:03.382665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:51:28.758641+0000","last_clean_scrub_stamp":"2024-04-18T21:53:03.382665+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:21.791354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378657,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378657,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41252,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330588,"statfs":{"total":96636764160,"available":96594522112,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378657,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378657,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411360,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2330624,"data_stored":1626428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41256,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330584,"statfs":{"total":96636764160,"available":96594518016,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411361,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411359,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2330624,"data_stored":1626428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444064,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":1097500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:53:52.993 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-18T21:53:19.115220+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=19, tm_wday=3, tm_yday=109, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=46, tm_wday=3, tm_yday=109, tm_isdst=0) 2024-04-18T21:53:52.993 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-18T21:53:00.433398+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=0, tm_wday=3, tm_yday=109, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=46, tm_wday=3, tm_yday=109, tm_isdst=0) 2024-04-18T21:53:52.994 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-18T21:52:59.203898+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=52, tm_sec=59, tm_wday=3, tm_yday=109, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=46, tm_wday=3, tm_yday=109, tm_isdst=0) 2024-04-18T21:53:52.994 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-18T21:53:04.014520+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=4, tm_wday=3, tm_yday=109, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=46, tm_wday=3, tm_yday=109, tm_isdst=0) 2024-04-18T21:53:52.994 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-18T21:53:13.569707+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=13, tm_wday=3, tm_yday=109, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=46, tm_wday=3, tm_yday=109, tm_isdst=0) 2024-04-18T21:53:52.994 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-18T21:53:00.139489+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=0, tm_wday=3, tm_yday=109, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=46, tm_wday=3, tm_yday=109, tm_isdst=0) 2024-04-18T21:53:52.994 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-18T21:53:03.382665+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=3, tm_wday=3, tm_yday=109, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=21, tm_min=53, tm_sec=46, tm_wday=3, tm_yday=109, tm_isdst=0) 2024-04-18T21:53:52.994 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-18T21:54:12.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T21:54:13.307 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-18T21:54:13.308 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-18T21:54:13.325 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-04-18T21:54:12.297670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":447044,"kb_used_data":22964,"kb_used_omap":18,"kb_used_meta":423917,"kb_avail":1132015036,"statfs":{"total":1159641169920,"available":1159183396864,"internally_reserved":0,"allocated":23515136,"data_stored":14778674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19139,"internal_metadata":434091325},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.905430"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":292,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-18T21:54:11.589790+0000","last_change":"2024-04-18T21:53:50.968949+0000","last_active":"2024-04-18T21:54:11.589790+0000","last_peered":"2024-04-18T21:54:11.589790+0000","last_clean":"2024-04-18T21:54:11.589790+0000","last_became_active":"2024-04-18T21:53:17.481925+0000","last_became_peered":"2024-04-18T21:53:17.481925+0000","last_unstale":"2024-04-18T21:54:11.589790+0000","last_undegraded":"2024-04-18T21:54:11.589790+0000","last_fullsized":"2024-04-18T21:54:11.589790+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:50.968699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:50.968699+0000","last_clean_scrub_stamp":"2024-04-18T21:53:50.968699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:52.238958+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":291,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-18T21:54:08.645598+0000","last_change":"2024-04-18T21:53:52.968416+0000","last_active":"2024-04-18T21:54:08.645598+0000","last_peered":"2024-04-18T21:54:08.645598+0000","last_clean":"2024-04-18T21:54:08.645598+0000","last_became_active":"2024-04-18T21:53:17.485150+0000","last_became_peered":"2024-04-18T21:53:17.485150+0000","last_unstale":"2024-04-18T21:54:08.645598+0000","last_undegraded":"2024-04-18T21:54:08.645598+0000","last_fullsized":"2024-04-18T21:54:08.645598+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:52.968015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:52.968015+0000","last_clean_scrub_stamp":"2024-04-18T21:53:52.968015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:07.129924+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":281,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-18T21:54:08.510591+0000","last_change":"2024-04-18T21:53:49.674275+0000","last_active":"2024-04-18T21:54:08.510591+0000","last_peered":"2024-04-18T21:54:08.510591+0000","last_clean":"2024-04-18T21:54:08.510591+0000","last_became_active":"2024-04-18T21:51:33.299371+0000","last_became_peered":"2024-04-18T21:51:33.299371+0000","last_unstale":"2024-04-18T21:54:08.510591+0000","last_undegraded":"2024-04-18T21:54:08.510591+0000","last_fullsized":"2024-04-18T21:54:08.510591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:49.674156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:49.674156+0000","last_clean_scrub_stamp":"2024-04-18T21:53:49.674156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:16.819592+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-18T21:54:08.440068+0000","last_change":"2024-04-18T21:53:53.306979+0000","last_active":"2024-04-18T21:54:08.440068+0000","last_peered":"2024-04-18T21:54:08.440068+0000","last_clean":"2024-04-18T21:54:08.440068+0000","last_became_active":"2024-04-18T21:52:40.515921+0000","last_became_peered":"2024-04-18T21:52:40.515921+0000","last_unstale":"2024-04-18T21:54:08.440068+0000","last_undegraded":"2024-04-18T21:54:08.440068+0000","last_fullsized":"2024-04-18T21:54:08.440068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:53.306741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:53.306741+0000","last_clean_scrub_stamp":"2024-04-18T21:53:53.306741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:08.537604+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":282,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-18T21:54:08.664337+0000","last_change":"2024-04-18T21:53:51.120858+0000","last_active":"2024-04-18T21:54:08.664337+0000","last_peered":"2024-04-18T21:54:08.664337+0000","last_clean":"2024-04-18T21:54:08.664337+0000","last_became_active":"2024-04-18T21:51:33.011112+0000","last_became_peered":"2024-04-18T21:51:33.011112+0000","last_unstale":"2024-04-18T21:54:08.664337+0000","last_undegraded":"2024-04-18T21:54:08.664337+0000","last_fullsized":"2024-04-18T21:54:08.664337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:51.120610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:51.120610+0000","last_clean_scrub_stamp":"2024-04-18T21:53:51.120610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:15.214461+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":291,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-18T21:54:08.645566+0000","last_change":"2024-04-18T21:53:53.803818+0000","last_active":"2024-04-18T21:54:08.645566+0000","last_peered":"2024-04-18T21:54:08.645566+0000","last_clean":"2024-04-18T21:54:08.645566+0000","last_became_active":"2024-04-18T21:52:40.414356+0000","last_became_peered":"2024-04-18T21:52:40.414356+0000","last_unstale":"2024-04-18T21:54:08.645566+0000","last_undegraded":"2024-04-18T21:54:08.645566+0000","last_fullsized":"2024-04-18T21:54:08.645566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:53.803594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:53.803594+0000","last_clean_scrub_stamp":"2024-04-18T21:53:53.803594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:15.395324+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-18T21:54:08.519779+0000","last_change":"2024-04-18T21:53:51.689505+0000","last_active":"2024-04-18T21:54:08.519779+0000","last_peered":"2024-04-18T21:54:08.519779+0000","last_clean":"2024-04-18T21:54:08.519779+0000","last_became_active":"2024-04-18T21:53:16.620879+0000","last_became_peered":"2024-04-18T21:53:16.620879+0000","last_unstale":"2024-04-18T21:54:08.519779+0000","last_undegraded":"2024-04-18T21:54:08.519779+0000","last_fullsized":"2024-04-18T21:54:08.519779+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:51.689236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:51.689236+0000","last_clean_scrub_stamp":"2024-04-18T21:53:51.689236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:10.720291+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":292,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-18T21:54:08.664348+0000","last_change":"2024-04-18T21:53:52.245105+0000","last_active":"2024-04-18T21:54:08.664348+0000","last_peered":"2024-04-18T21:54:08.664348+0000","last_clean":"2024-04-18T21:54:08.664348+0000","last_became_active":"2024-04-18T21:52:40.513856+0000","last_became_peered":"2024-04-18T21:52:40.513856+0000","last_unstale":"2024-04-18T21:54:08.664348+0000","last_undegraded":"2024-04-18T21:54:08.664348+0000","last_fullsized":"2024-04-18T21:54:08.664348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T21:53:52.244878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T21:53:52.244878+0000","last_clean_scrub_stamp":"2024-04-18T21:53:52.244878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:02.817717+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"144'139","reported_seq":370,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-18T21:54:08.419620+0000","last_change":"2024-04-18T21:53:52.232953+0000","last_active":"2024-04-18T21:54:08.419620+0000","last_peered":"2024-04-18T21:54:08.419620+0000","last_clean":"2024-04-18T21:54:08.419620+0000","last_became_active":"2024-04-18T21:53:20.654459+0000","last_became_peered":"2024-04-18T21:53:20.654459+0000","last_unstale":"2024-04-18T21:54:08.419620+0000","last_undegraded":"2024-04-18T21:54:08.419620+0000","last_fullsized":"2024-04-18T21:54:08.419620+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'139","last_scrub_stamp":"2024-04-18T21:53:52.232680+0000","last_deep_scrub":"144'139","last_deep_scrub_stamp":"2024-04-18T21:53:52.232680+0000","last_clean_scrub_stamp":"2024-04-18T21:53:52.232680+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:19.793166+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":19,"seq":81604378661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1871872,"data_stored":1144048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41340,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330500,"statfs":{"total":96636764160,"available":96594432000,"internally_reserved":0,"allocated":1884160,"data_stored":1151806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37168,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1871872,"data_stored":1144048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":19,"seq":81604378661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1871872,"data_stored":1144048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1871872,"data_stored":1144048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411364,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1871872,"data_stored":1144048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33528,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338312,"statfs":{"total":96636764160,"available":96602431488,"internally_reserved":0,"allocated":2404352,"data_stored":1672976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41328,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330512,"statfs":{"total":96636764160,"available":96594444288,"internally_reserved":0,"allocated":1871872,"data_stored":1144048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37168,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1871872,"data_stored":1144048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":18,"seq":77309411363,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33528,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338312,"statfs":{"total":96636764160,"available":96602431488,"internally_reserved":0,"allocated":2404352,"data_stored":1672976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":17,"seq":73014444068,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37156,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334684,"statfs":{"total":96636764160,"available":96598716416,"internally_reserved":0,"allocated":1859584,"data_stored":1136290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":17,"seq":73014444068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37156,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334684,"statfs":{"total":96636764160,"available":96598716416,"internally_reserved":0,"allocated":1859584,"data_stored":1136290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T21:54:13.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-18T21:54:13.646 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-18T21:54:13.646 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-18T21:54:13.647 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-18T21:54:13.647 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:13.732 INFO:tasks.ceph.osd.0:Stopped 2024-04-18T21:54:13.733 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-18T21:54:13.733 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:13.833 INFO:tasks.ceph.osd.1:Stopped 2024-04-18T21:54:13.834 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-18T21:54:13.834 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:19.935 INFO:tasks.ceph.osd.2:Stopped 2024-04-18T21:54:19.935 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-18T21:54:19.935 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:20.013 INFO:tasks.ceph.osd.3:Stopped 2024-04-18T21:54:20.013 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-18T21:54:20.013 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:20.101 INFO:tasks.ceph.osd.4:Stopped 2024-04-18T21:54:20.101 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-18T21:54:20.101 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:20.191 INFO:tasks.ceph.osd.5:Stopped 2024-04-18T21:54:20.191 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-18T21:54:20.191 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:20.272 INFO:tasks.ceph.osd.6:Stopped 2024-04-18T21:54:20.272 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-18T21:54:20.272 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:20.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-18T21:54:26.374 INFO:tasks.ceph.osd.7:Stopped 2024-04-18T21:54:26.374 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-04-18T21:54:26.374 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:26.453 INFO:tasks.ceph.osd.8:Stopped 2024-04-18T21:54:26.453 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-04-18T21:54:26.453 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:26.546 INFO:tasks.ceph.osd.9:Stopped 2024-04-18T21:54:26.546 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-04-18T21:54:26.546 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:26.634 INFO:tasks.ceph.osd.10:Stopped 2024-04-18T21:54:26.634 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-04-18T21:54:26.634 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:26.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-18T21:54:26.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-18T21:54:26.732 INFO:tasks.ceph.osd.11:Stopped 2024-04-18T21:54:26.732 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-18T21:54:26.733 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-18T21:54:26.733 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:26.762 INFO:tasks.ceph.mgr.x:Stopped 2024-04-18T21:54:26.763 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-18T21:54:26.763 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-18T21:54:26.763 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:26.784 INFO:tasks.ceph.mon.a:Stopped 2024-04-18T21:54:26.784 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-18T21:54:26.784 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:26.827 INFO:tasks.ceph.mon.b:Stopped 2024-04-18T21:54:26.827 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-18T21:54:26.827 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T21:54:26.845 INFO:tasks.ceph.mon.c:Stopped 2024-04-18T21:54:26.846 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-18T21:54:26.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-04-18T21:54:26.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:54:26.864 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-18T21:54:27.289 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:54:27.289 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-18T21:54:27.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:54:27.471 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-18T21:54:27.654 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:54:27.654 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-18T21:54:27.846 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:54:27.847 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-18T21:54:28.159 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:54:28.159 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-18T21:54:28.348 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:54:28.349 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-18T21:54:28.550 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:54:28.550 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-18T21:54:28.737 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:54:28.738 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-04-18T21:54:29.136 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:54:29.137 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-04-18T21:54:29.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:54:29.334 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-04-18T21:54:29.509 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:54:29.509 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-04-18T21:54:29.676 INFO:tasks.ceph:Archiving mon data... 2024-04-18T21:54:29.677 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/data/mon.a.tgz 2024-04-18T21:54:29.677 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-04-18T21:54:29.683 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.cTmIN6ccuu 2024-04-18T21:54:29.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.cTmIN6ccuu 2024-04-18T21:54:29.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.cTmIN6ccuu 2024-04-18T21:54:29.881 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.cTmIN6ccuu is 368KB 2024-04-18T21:54:29.943 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.cTmIN6ccuu 2024-04-18T21:54:29.948 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/data/mon.b.tgz 2024-04-18T21:54:29.948 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-04-18T21:54:29.954 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.WsRKjGc729 2024-04-18T21:54:29.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.WsRKjGc729 2024-04-18T21:54:30.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.WsRKjGc729 2024-04-18T21:54:30.189 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.WsRKjGc729 is 507KB 2024-04-18T21:54:30.251 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.WsRKjGc729 2024-04-18T21:54:30.257 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/data/mon.c.tgz 2024-04-18T21:54:30.257 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-04-18T21:54:30.263 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.JSlSMZXA26 2024-04-18T21:54:30.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.JSlSMZXA26 2024-04-18T21:54:30.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.JSlSMZXA26 2024-04-18T21:54:30.497 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.JSlSMZXA26 is 506KB 2024-04-18T21:54:30.559 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.JSlSMZXA26 2024-04-18T21:54:30.564 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-18T21:54:30.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-18T21:54:30.567 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-18T21:54:30.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-18T21:54:30.629 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-18T21:54:30.634 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-18T21:54:30.665 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-18T21:54:30.665 INFO:tasks.ceph:Archiving crash dumps... 2024-04-18T21:54:30.667 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi097/crash 2024-04-18T21:54:30.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-18T21:54:30.689 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi104/crash 2024-04-18T21:54:30.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-18T21:54:30.703 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi183/crash 2024-04-18T21:54:30.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-18T21:54:30.721 INFO:tasks.ceph:Compressing logs... 2024-04-18T21:54:30.721 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-18T21:54:30.735 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-18T21:54:30.750 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-18T21:54:30.754 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30591.log 2024-04-18T21:54:30.755 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35339.log 2024-04-18T21:54:30.755 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31511.log 2024-04-18T21:54:30.755 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30591.log.gz 2024-04-18T21:54:30.756 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35984.log 2024-04-18T21:54:30.756 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35339.log.gz 2024-04-18T21:54:30.756 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31511.log.gz 2024-04-18T21:54:30.756 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37206.log 2024-04-18T21:54:30.757 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31437.log 2024-04-18T21:54:30.758 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37648.log 2024-04-18T21:54:30.758 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37206.log.gz 2024-04-18T21:54:30.758 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35984.log.gz 2024-04-18T21:54:30.758 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31437.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31437.log.gz 2024-04-18T21:54:30.758 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33374.log 2024-04-18T21:54:30.758 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37648.log.gz 2024-04-18T21:54:30.758 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37518.log 2024-04-18T21:54:30.760 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31973.log 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33374.log.gz 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35046.log 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37518.log.gz 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36296.log 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31973.log.gz 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25249.log 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35046.log.gz 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32090.log 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36296.log.gz 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.25249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29889.log 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25249.log.gz 2024-04-18T21:54:30.761 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34646.log 2024-04-18T21:54:30.762 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32090.log.gz 2024-04-18T21:54:30.762 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35181.log 2024-04-18T21:54:30.762 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29889.log.gz 2024-04-18T21:54:30.766 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34646.log.gz 2024-04-18T21:54:30.766 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35338.log 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36302.log 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35181.log.gz 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29863.log 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35338.log.gz 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32393.log 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36302.log.gz 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32151.log 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29863.log.gz 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31358.log 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32393.log.gz 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32151.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32151.log.gz 2024-04-18T21:54:30.767 INFO:teuthology.orchestra.run.smithi097.stderr:gzip/var/log/ceph/ceph-client.admin.31358.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30288.log 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31358.log.gz 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31271.log 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.30288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32719.log 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30288.log.gz 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35485.log 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31271.log.gz 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36602.log 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32719.log.gz 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36286.log 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35485.log.gz 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34450.log 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36602.log.gz 2024-04-18T21:54:30.768 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36623.log 2024-04-18T21:54:30.769 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36286.log.gz 2024-04-18T21:54:30.769 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36626.log 2024-04-18T21:54:30.769 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34450.log.gz 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-18T21:54:30.786 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: 85.2% 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-18T21:54:30.787 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25091.log 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25211.log 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25091.log: /var/log/ceph/ceph-client.admin.25211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25091.log.gz 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25211.log.gz 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25307.log 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25259.log 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25307.log.gz 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25163.log 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-04-18T21:54:30.808 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25259.log.gz 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25163.log.gz 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25139.log 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25067.log 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25139.log.gz 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25067.log.gz 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25235.log 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25235.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25235.log.gz 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25187.log 2024-04-18T21:54:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25115.log 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25187.log: 85.2% 0.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25187.log.gz 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25043.log 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25283.log 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25115.log: 89.4% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25115.log.gz 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25043.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25043.log.gz 2024-04-18T21:54:30.810 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.25283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25283.log.gz 2024-04-18T21:54:30.821 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.36623.log: -- /var/log/ceph/ceph-client.admin.36062.log 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36623.log.gz 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36626.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36626.log.gz -5 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33098.log 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36894.log 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36062.log.gz 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31313.log 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33098.log.gz 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35672.log 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36894.log.gz 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34939.log 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31313.log.gz 2024-04-18T21:54:30.822 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34172.log 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35672.log.gz 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34939.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.34939.log.gz -- 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr: /var/log/ceph/ceph-client.admin.31878.log 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34172.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34172.log.gz -5 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32612.log 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33453.log 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31878.log.gz 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30303.log 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32612.log.gz 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37258.log 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33453.log.gz 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37934.log 2024-04-18T21:54:30.823 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30303.log.gz 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31061.log 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37258.log.gz 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32993.log 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37934.log.gz 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32039.log 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31061.log.gz 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32993.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32993.log.gz -5 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37544.log 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32039.log.gz 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37544.log.gz 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36166.log 2024-04-18T21:54:30.824 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33823.log 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35958.log 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36166.log.gz 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33823.log.gz 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34679.log 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33348.log 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35958.log.gz 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34679.log.gz 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30669.log 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33348.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33348.log.gz 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30669.log.gz 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38012.log 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30290.log 2024-04-18T21:54:30.825 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.38012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38012.log.gz 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35405.log 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31610.log 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30290.log.gz 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35405.log.gz 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30779.log 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31610.log.gz 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30011.log 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36967.log 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30779.log.gz 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30011.log.gz 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29998.log 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36572.log 2024-04-18T21:54:30.826 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36967.log.gz 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29998.log.gz 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33299.log 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37466.log 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36572.log: /var/log/ceph/ceph-client.admin.33299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36572.log.gz 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33299.log.gz 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36311.log 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37232.log 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37466.log.gz 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36311.log.gz 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35249.log 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30274.log 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37232.log.gz 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35249.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20250.log 2024-04-18T21:54:30.827 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35249.log.gz 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30274.log.gz 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35409.log 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37830.log 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.tmp-client.admin.20250.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20250.log.gz 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35409.log.gz 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36088.log 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30856.log 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37830.log: /var/log/ceph/ceph-client.admin.36088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37830.log.gz 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36088.log.gz 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37908.log 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36980.log 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30856.log.gz 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30851.log 2024-04-18T21:54:30.828 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37908.log.gz 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36980.log.gz 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36955.log 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30851.log.gz 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36955.log.gz 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33124.log 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37336.log 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35698.log: /var/log/ceph/ceph-client.admin.33124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29995.log 0.0% 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33124.log.gz 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31756.log 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37336.log.gz 2024-04-18T21:54:30.829 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36956.log 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29995.log.gz 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31756.log.gz 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25121.log 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36584.log 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36956.log.gz 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.25121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29812.log 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25121.log.gz 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36584.log.gz 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37674.log 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37622.log 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29812.log.gz 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33796.log 2024-04-18T21:54:30.830 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37674.log.gz 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37622.log.gz 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31170.log 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31765.log 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33796.log: /var/log/ceph/ceph-client.admin.31170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33796.log.gz 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31170.log.gz 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35802.log 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31765.log.gz 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34765.log 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35802.log: /var/log/ceph/ceph-osd.3.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35802.log.gz 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31685.log 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34765.log.gz 2024-04-18T21:54:30.831 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36313.log 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36612.log 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31685.log.gz 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36313.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.36594.log 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36313.log.gz 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35932.log 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36612.log.gz 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37440.log 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36594.log.gz 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35932.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35932.log.gz -5 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32909.log 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30328.log 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37440.log.gz 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32909.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-18T21:54:30.832 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32909.log.gz 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34097.log 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30328.log.gz 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38036.log 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36140.log 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34097.log.gz/var/log/ceph/ceph-client.admin.38036.log: 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.38036.log.gz --verbose 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr: -- /var/log/ceph/ceph-client.admin.36613.log 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36140.log.gz 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32562.log 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36314.log 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36613.log.gz 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36305.log 2024-04-18T21:54:30.833 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32562.log.gz 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36314.log.gz 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32345.log 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29989.log 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36305.log.gz 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35255.log 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32345.log.gz 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29989.log.gz 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32397.log 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30329.log 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35255.log.gz 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31412.log 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32397.log.gz 2024-04-18T21:54:30.834 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29988.log 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30329.log.gz 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36614.log 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31412.log.gz 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36976.log 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29988.log.gz/var/log/ceph/ceph-client.admin.36614.log: 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34422.log 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36614.log.gz 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32520.log 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36976.log.gz 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35223.log 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34422.log.gz 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32520.log.gz 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35102.log 2024-04-18T21:54:30.835 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35223.log.gz 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33128.log 0.0% 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35102.log.gz 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35511.log 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33128.log.gz 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37596.log 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36036.log 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35511.log.gz 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35011.log 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37596.log.gz 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36036.log.gz 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34814.log 2024-04-18T21:54:30.836 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32452.log 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35011.log.gz 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31912.log 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34814.log.gz 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33597.log 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32452.log.gz 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31912.log.gz 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33572.log 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37986.log 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33597.log.gz 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34370.log 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33572.log.gz 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37570.log 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37986.log.gz 2024-04-18T21:54:30.837 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34370.log.gz 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35151.log 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37882.log 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37570.log.gz 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36300.log 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35151.log.gz 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37882.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37882.log.gz 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30007.log 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36300.log.gz/var/log/ceph/ceph.log: 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34295.log 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37752.log 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30007.log: /var/log/ceph/ceph-client.admin.34295.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30007.log.gz 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34295.log.gz 2024-04-18T21:54:30.838 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35101.log 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30008.log 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr: 85.2% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37752.log.gz 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35101.log.gz 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32745.log 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33677.log 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30008.log.gz 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30857.log 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32745.log.gz 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33677.log.gz 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31561.log 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33072.log 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30857.log: /var/log/ceph/ceph-client.admin.31561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36244.log 2024-04-18T21:54:30.839 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30857.log.gz 2024-04-18T21:54:30.840 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31561.log.gz 2024-04-18T21:54:30.840 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33072.log.gz 2024-04-18T21:54:30.860 INFO:teuthology.orchestra.run.smithi104.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-18T21:54:30.876 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30010.log 2024-04-18T21:54:30.876 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32232.log 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36244.log.gz 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36958.log 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30010.log.gz 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35563.log 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32232.log.gz/var/log/ceph/ceph-client.admin.36958.log: 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37726.log 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36958.log.gz 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37778.log 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35563.log: /var/log/ceph/ceph-client.admin.37726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35563.log.gz 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30317.log 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37726.log.gz 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37778.log.gz 2024-04-18T21:54:30.877 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35072.log 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29837.log 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30317.log.gz 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36218.log 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35072.log.gz 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35776.log 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29837.log: /var/log/ceph/ceph-client.admin.36218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29941.log 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29837.log.gz 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36218.log.gz 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35776.log.gz 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35594.log 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34897.log 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29941.log.gz 2024-04-18T21:54:30.878 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37856.log 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35594.log.gz 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34320.log 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34897.log: /var/log/ceph/ceph-client.admin.37856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34897.log.gz 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37856.log.gz 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36589.log 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37960.log 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34320.log.gz 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34571.log 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36589.log.gz 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37960.log.gz 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34069.log 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30279.log 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34571.log.gz 2024-04-18T21:54:30.879 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33183.log 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34069.log.gz 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32830.log 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30279.log: /var/log/ceph/ceph-client.admin.33183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31111.log 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30279.log.gz 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33183.log.gz 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32830.log.gz 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36912.log 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29990.log 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31111.log.gz 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37362.log 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36912.log.gz 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31237.log 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29990.log.gz 2024-04-18T21:54:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36114.log 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37362.log.gz 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35750.log 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31237.log.gz 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30753.log 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36114.log: /var/log/ceph/ceph-client.admin.35750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35906.log 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36114.log.gz 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35750.log.gz 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30753.log.gz 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33944.log 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36898.log 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35906.log.gz 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34210.log 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33944.log.gz 2024-04-18T21:54:30.881 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36898.log: /var/log/ceph/ceph-client.admin.34210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29994.log 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36898.log.gz 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34210.log.gz 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32265.log 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.29994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31761.log 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29994.log.gz 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32265.log.gz 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36297.log 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35828.log 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31761.log.gz 2024-04-18T21:54:30.882 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36297.log.gz 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36192.log 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31116.log 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35828.log: /var/log/ceph/ceph-client.admin.36192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36957.log 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35828.log.gz 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36192.log.gz 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31116.log.gz 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35307.log 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31216.log 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36957.log: /var/log/ceph/ceph-client.admin.35307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32669.log 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35307.log.gz 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36957.log.gz 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.0.31216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.31216.log.gz 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33023.log 2024-04-18T21:54:30.883 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34575.log 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32669.log: /var/log/ceph/ceph-client.admin.33023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32695.log 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32669.log.gz 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33023.log.gz 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34575.log.gz 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35303.log 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33421.log 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32695.log: /var/log/ceph/ceph-client.admin.35303.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32695.log.gz -5 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33232.log 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35303.log.gz 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33421.log.gz 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30300.log 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35724.log 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33232.log.gz 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30300.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.30300.log.gz --verbose 2024-04-18T21:54:30.884 INFO:teuthology.orchestra.run.smithi097.stderr: -- /var/log/ceph/ceph-client.admin.30000.log 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36635.log 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35724.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30595.log 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30000.log: -- replaced with /var/log/ceph/ceph-client.admin.35724.log.gz 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30000.log.gz 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36635.log.gz 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33896.log 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35537.log 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30595.log.gz 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33896.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.33709.log 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33896.log.gz 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37388.log 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35537.log.gz 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32311.log 2024-04-18T21:54:30.885 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33709.log: /var/log/ceph/ceph-client.admin.37388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33709.log.gz 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33871.log 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37388.log.gz 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32311.log.gz 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33503.log 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36974.log 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33871.log.gz 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.33503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29915.log 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33503.log.gz 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32180.log 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36974.log: /var/log/ceph/ceph-client.admin.29915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34446.log 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29915.log.gz 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36974.log.gz/var/log/ceph/ceph-client.admin.32180.log: 2024-04-18T21:54:30.886 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32180.log.gz 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30721.log 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36299.log 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34446.log: gzip/var/log/ceph/ceph-client.admin.30721.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31635.log 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34446.log.gz 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30721.log.gz 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36299.log.gz 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31711.log 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36601.log 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31635.log.gz 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32771.log 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31711.log.gz 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33647.log 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31937.log 2024-04-18T21:54:30.887 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36601.log: /var/log/ceph/ceph-client.admin.32771.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32771.log.gz 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36601.log.gz/var/log/ceph/ceph-client.admin.33647.log: 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33647.log.gz 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37310.log 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36010.log 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31937.log.gz 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25036.log 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37310.log.gz 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36978.log 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36010.log: /var/log/ceph/ceph-client.admin.25036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34011.log 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36010.log.gz 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25036.log.gz 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32834.log 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36978.log.gz 2024-04-18T21:54:30.888 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30309.log 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34011.log: /var/log/ceph/ceph-client.admin.32834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32483.log 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34011.log.gz 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32834.log.gz 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30309.log.gz 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37284.log 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32483.log.gz 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32954.log 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30846.log 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37284.log: /var/log/ceph/ceph-client.admin.32954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35854.log 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37284.log.gz 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32954.log.gz 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30846.log.gz 2024-04-18T21:54:30.889 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36301.log 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30809.log 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35854.log: /var/log/ceph/ceph-client.admin.36301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32065.log 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35854.log.gz 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36301.log.gz 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30809.log.gz 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34146.log 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31970.log 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.32065.log: /var/log/ceph/ceph-client.admin.34146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37804.log 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32065.log.gz 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34146.log.gz 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.31970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30643.log 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31970.log.gz 2024-04-18T21:54:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36292.log 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37804.log: /var/log/ceph/ceph-client.admin.30643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30006.log 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30643.log.gz 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37804.log.gz 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36292.log.gz 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32341.log 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35646.log 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30006.log: /var/log/ceph/ceph-client.admin.32341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34848.log 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30006.log.gz 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32341.log.gz 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35646.log.gz 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31022.log 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30695.log 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34848.log: /var/log/ceph/ceph-client.admin.31022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37414.log 2024-04-18T21:54:30.891 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31022.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34848.log.gz 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.30695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30695.log.gz 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30307.log 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34396.log 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37414.log: /var/log/ceph/ceph-client.admin.30307.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37414.log.gz -- 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr: /var/log/ceph/ceph-client.admin.35459.log 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30307.log.gz 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34396.log.gz 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35620.log 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35880.log 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35459.log.gz 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35620.log.gz 2024-04-18T21:54:30.892 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35880.log.gz 2024-04-18T21:54:30.893 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.10.log: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-18T21:54:30.946 INFO:teuthology.orchestra.run.smithi104.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-18T21:54:31.139 INFO:teuthology.orchestra.run.smithi097.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-18T21:54:31.204 INFO:teuthology.orchestra.run.smithi097.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-18T21:54:31.503 INFO:teuthology.orchestra.run.smithi104.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-18T21:54:31.519 INFO:teuthology.orchestra.run.smithi183.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-04-18T21:54:31.535 INFO:teuthology.orchestra.run.smithi183.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-04-18T21:54:31.591 INFO:teuthology.orchestra.run.smithi183.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-04-18T21:54:31.658 INFO:teuthology.orchestra.run.smithi097.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-18T21:54:31.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-18T21:54:32.049 INFO:teuthology.orchestra.run.smithi183.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-04-18T21:54:32.051 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-04-18T21:54:32.051 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m1.278s 2024-04-18T21:54:32.051 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m3.534s 2024-04-18T21:54:32.051 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.125s 2024-04-18T21:54:32.102 INFO:teuthology.orchestra.run.smithi104.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-18T21:54:32.128 INFO:teuthology.orchestra.run.smithi097.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-18T21:54:32.142 INFO:teuthology.orchestra.run.smithi097.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-18T21:54:32.144 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-18T21:54:32.144 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m1.409s 2024-04-18T21:54:32.145 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m4.420s 2024-04-18T21:54:32.145 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m0.225s 2024-04-18T21:54:32.225 INFO:teuthology.orchestra.run.smithi104.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-18T21:54:32.280 INFO:teuthology.orchestra.run.smithi104.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-18T21:54:32.281 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-18T21:54:32.281 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m1.529s 2024-04-18T21:54:32.281 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m5.082s 2024-04-18T21:54:32.281 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.182s 2024-04-18T21:54:32.282 INFO:tasks.ceph:Archiving logs... 2024-04-18T21:54:32.283 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi097/log 2024-04-18T21:54:32.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-18T21:54:33.079 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi104/log 2024-04-18T21:54:33.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-18T21:54:33.765 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi183/log 2024-04-18T21:54:33.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-18T21:54:34.267 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-18T21:54:34.279 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-18T21:54:34.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-18T21:54:34.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-18T21:54:34.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-18T21:54:34.384 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-18T21:54:34.393 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-18T21:54:34.393 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T21:54:34.396 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T21:54:34.398 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T21:54:34.406 INFO:teuthology.orchestra.run.smithi097.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T21:54:34.406 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================== 2024-04-18T21:54:34.406 INFO:teuthology.orchestra.run.smithi097.stdout:+hv01.front.sepi 138.236.128.36 3 u 35 64 77 0.090 -2.126 0.913 2024-04-18T21:54:34.406 INFO:teuthology.orchestra.run.smithi097.stdout:+hv02.front.sepi 63.231.80.2 3 u 35 64 77 0.074 +0.066 0.925 2024-04-18T21:54:34.406 INFO:teuthology.orchestra.run.smithi097.stdout:*hv03.front.sepi 74.6.168.72 3 u 42 64 77 0.069 -0.686 1.196 2024-04-18T21:54:34.406 INFO:teuthology.orchestra.run.smithi097.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:54:34.408 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T21:54:34.409 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2024-04-18T21:54:34.409 INFO:teuthology.orchestra.run.smithi104.stdout:+hv01.front.sepi 138.236.128.36 3 u 39 64 77 0.128 -1.249 0.274 2024-04-18T21:54:34.409 INFO:teuthology.orchestra.run.smithi104.stdout:*hv02.front.sepi 63.231.80.2 3 u 36 64 77 0.107 +0.050 0.396 2024-04-18T21:54:34.409 INFO:teuthology.orchestra.run.smithi104.stdout:+hv03.front.sepi 74.6.168.72 3 u 37 64 77 0.104 +0.640 0.398 2024-04-18T21:54:34.409 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:54:34.412 INFO:teuthology.orchestra.run.smithi183.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T21:54:34.412 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================== 2024-04-18T21:54:34.412 INFO:teuthology.orchestra.run.smithi183.stdout:+hv01.front.sepi 138.236.128.36 3 u 37 64 77 0.129 -1.911 1.824 2024-04-18T21:54:34.412 INFO:teuthology.orchestra.run.smithi183.stdout:*hv02.front.sepi 63.231.80.2 3 u 35 64 77 0.101 -0.674 1.953 2024-04-18T21:54:34.412 INFO:teuthology.orchestra.run.smithi183.stdout:+hv03.front.sepi 74.6.168.72 3 u 29 64 77 0.130 -0.019 1.789 2024-04-18T21:54:34.412 INFO:teuthology.orchestra.run.smithi183.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T21:54:34.413 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-18T21:54:34.422 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-18T21:54:34.423 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-18T21:54:34.473 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-18T21:54:34.524 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-18T21:54:34.574 INFO:teuthology.task.internal:Duration was 808.201198 seconds 2024-04-18T21:54:34.575 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-18T21:54:34.624 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-18T21:54:34.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-18T21:54:34.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-18T21:54:34.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-18T21:54:34.668 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-18T21:54:34.668 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-04-18T21:54:34.668 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-18T21:54:34.717 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-04-18T21:54:34.718 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-18T21:54:34.728 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-04-18T21:54:34.729 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-18T21:54:34.739 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-18T21:54:34.740 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-18T21:54:34.766 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-18T21:54:34.774 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-18T21:54:34.807 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-18T21:54:34.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-18T21:54:34.833 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-18T21:54:34.842 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-18T21:54:34.907 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-18T21:54:34.918 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-18T21:54:34.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-18T21:54:34.930 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-18T21:54:34.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-18T21:54:34.953 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-18T21:54:34.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-18T21:54:34.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-18T21:54:34.983 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-04-18T21:54:34.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-18T21:54:34.995 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-04-18T21:54:35.005 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-04-18T21:54:35.052 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-18T21:54:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:54:35.100 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-18T21:54:35.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:54:35.106 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-18T21:54:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T21:54:35.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-18T21:54:35.121 INFO:teuthology.task.internal:Transferring archived files... 2024-04-18T21:54:35.122 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi097 2024-04-18T21:54:35.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-18T21:54:35.165 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi104 2024-04-18T21:54:35.165 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-18T21:54:35.187 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662549/remote/smithi183 2024-04-18T21:54:35.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-18T21:54:35.207 INFO:teuthology.task.internal:Removing archive directory... 2024-04-18T21:54:35.208 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-18T21:54:35.210 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-18T21:54:35.226 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-18T21:54:35.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-18T21:54:35.265 INFO:teuthology.task.internal:Not uploading archives. 2024-04-18T21:54:35.265 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-18T21:54:35.300 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-18T21:54:35.300 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-18T21:54:35.303 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-18T21:54:35.305 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-18T21:54:35.306 INFO:teuthology.orchestra.run.smithi097.stdout: 658642 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 18 21:54 /home/ubuntu/cephtest 2024-04-18T21:54:35.309 INFO:teuthology.orchestra.run.smithi104.stdout: 658633 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 18 21:54 /home/ubuntu/cephtest 2024-04-18T21:54:35.310 INFO:teuthology.orchestra.run.smithi183.stdout: 658463 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 18 21:54 /home/ubuntu/cephtest 2024-04-18T21:54:35.311 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-18T21:54:35.377 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-18T21:54:35.478 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} duration: 808.2011983394623 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-18T21:54:35.478 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T21:54:35.558 INFO:teuthology.run:pass